2026-02-20T22:21:26.816 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-20T22:21:26.832 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T22:21:26.913 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587 branch: wip-pdonnell-testing-20260219.182737-tentacle description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '61587' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial no_nested_subset: false os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-pdonnell-testing-20260219.182737-tentacle ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: f7870454149f22574086dd2ff55d050585136531 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f7870454149f22574086dd2ff55d050585136531 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20260219.182737-tentacle sha1: f7870454149f22574086dd2ff55d050585136531 owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 7161 sha1: f7870454149f22574086dd2ff55d050585136531 sleep_before_teardown: 0 subset: 111/120000 suite: rbd suite_branch: wip-pdonnell-testing-20260219.182737-tentacle suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: f7870454149f22574086dd2ff55d050585136531 targets: trial032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZKsPo+lvwLxn7Ee212rieC6zWJB+5QbSZDG9k7krFYPsVwEopXdvw5hQt2KBegIIEdhoyaRHkkH4ZdwY0vIO0= trial107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOh8ioD937Osrcchl2HKEKdPFRr9QLyyACeqMcp5QXhMDbgV03LgBME8Q//ctKqZ6OIYcKsXBxkwhD+hdFR1J9c= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-20_16:15:01 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699 2026-02-20T22:21:26.913 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa; will attempt to use it 2026-02-20T22:21:26.913 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa/tasks 2026-02-20T22:21:26.913 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-20T22:21:26.915 INFO:teuthology.task.internal:Checking packages... 2026-02-20T22:21:26.915 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f7870454149f22574086dd2ff55d050585136531' 2026-02-20T22:21:26.915 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-20T22:21:26.915 INFO:teuthology.packaging:ref: None 2026-02-20T22:21:26.915 INFO:teuthology.packaging:tag: None 2026-02-20T22:21:26.915 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20260219.182737-tentacle 2026-02-20T22:21:26.915 INFO:teuthology.packaging:sha1: f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:21:26.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-pdonnell-testing-20260219.182737-tentacle 2026-02-20T22:21:27.035 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-677-gf7870454-1jammy 2026-02-20T22:21:27.036 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-20T22:21:27.047 INFO:teuthology.task.internal:no buildpackages task found 2026-02-20T22:21:27.048 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-20T22:21:27.061 INFO:teuthology.task.internal:Saving configuration 2026-02-20T22:21:27.075 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-20T22:21:27.087 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-20T22:21:27.164 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial032.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 22:17:57.055357', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHarVpJNzLkaTtWS6VBNv7P0WNITB7Xy1lvy778k5SHG'} 2026-02-20T22:21:27.219 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial107.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-20 22:17:57.054742', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIy+tjHgf2/kZBP39XTC9KVTXRJdid1nKNcnqxEBZH00'} 2026-02-20T22:21:27.219 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-20T22:21:27.231 INFO:teuthology.task.internal:roles: ubuntu@trial032.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2026-02-20T22:21:27.231 INFO:teuthology.task.internal:roles: ubuntu@trial107.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2026-02-20T22:21:27.231 INFO:teuthology.run_tasks:Running task console_log... 2026-02-20T22:21:27.365 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x74c3c382d6c0>, signals=[15]) 2026-02-20T22:21:27.365 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-20T22:21:27.376 INFO:teuthology.task.internal:Opening connections... 2026-02-20T22:21:27.376 DEBUG:teuthology.task.internal:connecting to ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:21:27.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T22:21:27.440 DEBUG:teuthology.task.internal:connecting to ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:21:27.440 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T22:21:27.501 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-20T22:21:27.512 DEBUG:teuthology.orchestra.run.trial032:> uname -m 2026-02-20T22:21:27.516 INFO:teuthology.orchestra.run.trial032.stdout:x86_64 2026-02-20T22:21:27.516 DEBUG:teuthology.orchestra.run.trial032:> cat /etc/os-release 2026-02-20T22:21:27.559 INFO:teuthology.orchestra.run.trial032.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:NAME="Ubuntu" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:VERSION_ID="22.04" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:VERSION_CODENAME=jammy 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:ID=ubuntu 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:ID_LIKE=debian 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-20T22:21:27.560 INFO:teuthology.orchestra.run.trial032.stdout:UBUNTU_CODENAME=jammy 2026-02-20T22:21:27.560 INFO:teuthology.lock.ops:Updating trial032.front.sepia.ceph.com on lock server 2026-02-20T22:21:27.624 DEBUG:teuthology.orchestra.run.trial107:> uname -m 2026-02-20T22:21:27.627 INFO:teuthology.orchestra.run.trial107.stdout:x86_64 2026-02-20T22:21:27.628 DEBUG:teuthology.orchestra.run.trial107:> cat /etc/os-release 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:NAME="Ubuntu" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:VERSION_ID="22.04" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:VERSION_CODENAME=jammy 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:ID=ubuntu 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:ID_LIKE=debian 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-20T22:21:27.672 INFO:teuthology.orchestra.run.trial107.stdout:UBUNTU_CODENAME=jammy 2026-02-20T22:21:27.672 INFO:teuthology.lock.ops:Updating trial107.front.sepia.ceph.com on lock server 2026-02-20T22:21:27.733 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-20T22:21:27.745 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-20T22:21:27.757 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-20T22:21:27.757 DEBUG:teuthology.orchestra.run.trial032:> test '!' -e /home/ubuntu/cephtest 2026-02-20T22:21:27.758 DEBUG:teuthology.orchestra.run.trial107:> test '!' -e /home/ubuntu/cephtest 2026-02-20T22:21:27.762 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-20T22:21:27.774 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-20T22:21:27.774 DEBUG:teuthology.orchestra.run.trial032:> test -z $(ls -A /var/lib/ceph) 2026-02-20T22:21:27.806 DEBUG:teuthology.orchestra.run.trial107:> test -z $(ls -A /var/lib/ceph) 2026-02-20T22:21:27.809 INFO:teuthology.orchestra.run.trial032.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-20T22:21:27.813 INFO:teuthology.orchestra.run.trial107.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-20T22:21:27.814 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-20T22:21:27.967 INFO:teuthology.run_tasks:Running task kernel... 2026-02-20T22:21:27.986 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-20T22:21:27.987 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'cluster2.mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-20T22:21:27.987 DEBUG:teuthology.orchestra.run.trial032:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T22:21:27.987 DEBUG:teuthology.orchestra.run.trial107:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-20T22:21:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:21:27.990 DEBUG:teuthology.orchestra.run.trial032:> uname -r 2026-02-20T22:21:27.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:21:27.991 DEBUG:teuthology.orchestra.run.trial107:> uname -r 2026-02-20T22:21:28.036 INFO:teuthology.orchestra.run.trial107.stdout:5.15.0-170-generic 2026-02-20T22:21:28.036 INFO:teuthology.task.kernel:Running kernel on trial107: 5.15.0-170-generic 2026-02-20T22:21:28.036 DEBUG:teuthology.orchestra.run.trial107:> sudo apt-get clean 2026-02-20T22:21:28.040 INFO:teuthology.orchestra.run.trial032.stdout:5.15.0-170-generic 2026-02-20T22:21:28.040 INFO:teuthology.task.kernel:Running kernel on trial032: 5.15.0-170-generic 2026-02-20T22:21:28.040 DEBUG:teuthology.orchestra.run.trial032:> sudo apt-get clean 2026-02-20T22:21:28.095 DEBUG:teuthology.orchestra.run.trial107:> sudo apt-get update 2026-02-20T22:21:28.098 DEBUG:teuthology.orchestra.run.trial032:> sudo apt-get update 2026-02-20T22:21:28.353 INFO:teuthology.orchestra.run.trial032.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T22:21:28.358 INFO:teuthology.orchestra.run.trial107.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T22:21:28.359 INFO:teuthology.orchestra.run.trial032.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-20T22:21:28.365 INFO:teuthology.orchestra.run.trial107.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-20T22:21:28.393 INFO:teuthology.orchestra.run.trial032.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-20T22:21:28.403 INFO:teuthology.orchestra.run.trial107.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-20T22:21:28.407 INFO:teuthology.orchestra.run.trial032.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-20T22:21:28.418 INFO:teuthology.orchestra.run.trial107.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-20T22:21:28.494 INFO:teuthology.orchestra.run.trial032.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-20T22:21:28.503 INFO:teuthology.orchestra.run.trial107.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3245 kB] 2026-02-20T22:21:28.550 INFO:teuthology.orchestra.run.trial032.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-20T22:21:28.555 INFO:teuthology.orchestra.run.trial032.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-20T22:21:28.556 INFO:teuthology.orchestra.run.trial032.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-20T22:21:28.563 INFO:teuthology.orchestra.run.trial032.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-20T22:21:28.565 INFO:teuthology.orchestra.run.trial032.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-20T22:21:28.566 INFO:teuthology.orchestra.run.trial032.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-20T22:21:28.569 INFO:teuthology.orchestra.run.trial107.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [495 kB] 2026-02-20T22:21:28.573 INFO:teuthology.orchestra.run.trial107.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.1 kB] 2026-02-20T22:21:28.574 INFO:teuthology.orchestra.run.trial107.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-20T22:21:28.583 INFO:teuthology.orchestra.run.trial107.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-20T22:21:28.585 INFO:teuthology.orchestra.run.trial107.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-20T22:21:28.586 INFO:teuthology.orchestra.run.trial107.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5310 kB] 2026-02-20T22:21:28.593 INFO:teuthology.orchestra.run.trial032.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-20T22:21:28.596 INFO:teuthology.orchestra.run.trial032.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-20T22:21:28.605 INFO:teuthology.orchestra.run.trial032.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-20T22:21:28.607 INFO:teuthology.orchestra.run.trial032.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-20T22:21:28.607 INFO:teuthology.orchestra.run.trial032.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-20T22:21:28.612 INFO:teuthology.orchestra.run.trial032.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-20T22:21:28.612 INFO:teuthology.orchestra.run.trial032.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-20T22:21:28.613 INFO:teuthology.orchestra.run.trial107.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1005 kB] 2026-02-20T22:21:28.615 INFO:teuthology.orchestra.run.trial107.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2983 kB] 2026-02-20T22:21:28.624 INFO:teuthology.orchestra.run.trial107.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [427 kB] 2026-02-20T22:21:28.626 INFO:teuthology.orchestra.run.trial107.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-20T22:21:28.626 INFO:teuthology.orchestra.run.trial107.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1018 kB] 2026-02-20T22:21:28.630 INFO:teuthology.orchestra.run.trial107.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.7 kB] 2026-02-20T22:21:28.630 INFO:teuthology.orchestra.run.trial107.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5147 kB] 2026-02-20T22:21:28.635 INFO:teuthology.orchestra.run.trial032.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-20T22:21:28.647 INFO:teuthology.orchestra.run.trial107.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [979 kB] 2026-02-20T22:21:32.469 INFO:teuthology.orchestra.run.trial032.stdout:Fetched 22.6 MB in 2s (13.1 MB/s) 2026-02-20T22:21:32.477 INFO:teuthology.orchestra.run.trial107.stdout:Fetched 22.6 MB in 2s (13.0 MB/s) 2026-02-20T22:21:32.997 INFO:teuthology.orchestra.run.trial032.stdout:Reading package lists... 2026-02-20T22:21:33.001 INFO:teuthology.orchestra.run.trial107.stdout:Reading package lists... 2026-02-20T22:21:33.010 DEBUG:teuthology.orchestra.run.trial032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-20T22:21:33.015 DEBUG:teuthology.orchestra.run.trial107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-20T22:21:33.043 INFO:teuthology.orchestra.run.trial032.stdout:Reading package lists... 2026-02-20T22:21:33.048 INFO:teuthology.orchestra.run.trial107.stdout:Reading package lists... 2026-02-20T22:21:33.154 INFO:teuthology.orchestra.run.trial032.stdout:Building dependency tree... 2026-02-20T22:21:33.155 INFO:teuthology.orchestra.run.trial032.stdout:Reading state information... 2026-02-20T22:21:33.155 INFO:teuthology.orchestra.run.trial107.stdout:Building dependency tree... 2026-02-20T22:21:33.155 INFO:teuthology.orchestra.run.trial107.stdout:Reading state information... 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout:linux-image-generic set to manually installed. 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout: sg3-utils-udev 2026-02-20T22:21:33.241 INFO:teuthology.orchestra.run.trial107.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T22:21:33.242 INFO:teuthology.orchestra.run.trial032.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-20T22:21:33.242 INFO:teuthology.orchestra.run.trial032.stdout:linux-image-generic set to manually installed. 2026-02-20T22:21:33.243 INFO:teuthology.orchestra.run.trial032.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T22:21:33.243 INFO:teuthology.orchestra.run.trial032.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T22:21:33.243 INFO:teuthology.orchestra.run.trial032.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T22:21:33.243 INFO:teuthology.orchestra.run.trial032.stdout: sg3-utils-udev 2026-02-20T22:21:33.243 INFO:teuthology.orchestra.run.trial032.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T22:21:33.259 INFO:teuthology.orchestra.run.trial107.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T22:21:33.262 DEBUG:teuthology.orchestra.run.trial107:> dpkg -s linux-image-generic 2026-02-20T22:21:33.262 INFO:teuthology.orchestra.run.trial032.stdout:0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T22:21:33.264 DEBUG:teuthology.orchestra.run.trial032:> dpkg -s linux-image-generic 2026-02-20T22:21:33.314 INFO:teuthology.orchestra.run.trial032.stdout:Package: linux-image-generic 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Status: install ok installed 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Priority: optional 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Section: kernel 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Installed-Size: 21 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Maintainer: Ubuntu Kernel Team 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Architecture: amd64 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Source: linux-meta 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Version: 5.15.0.170.159 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-20T22:21:33.315 INFO:teuthology.orchestra.run.trial032.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-20T22:21:33.316 INFO:teuthology.orchestra.run.trial032.stdout:Recommends: thermald 2026-02-20T22:21:33.316 INFO:teuthology.orchestra.run.trial032.stdout:Description: Generic Linux kernel image 2026-02-20T22:21:33.316 INFO:teuthology.orchestra.run.trial032.stdout: This package will always depend on the latest generic kernel image 2026-02-20T22:21:33.316 INFO:teuthology.orchestra.run.trial032.stdout: available. 2026-02-20T22:21:33.316 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T22:21:33.316 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2026-02-20T22:21:33.317 DEBUG:teuthology.orchestra.run.trial032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Package: linux-image-generic 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Status: install ok installed 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Priority: optional 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Section: kernel 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Installed-Size: 21 2026-02-20T22:21:33.318 INFO:teuthology.orchestra.run.trial107.stdout:Maintainer: Ubuntu Kernel Team 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Architecture: amd64 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Source: linux-meta 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Version: 5.15.0.170.159 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Recommends: thermald 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout:Description: Generic Linux kernel image 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout: This package will always depend on the latest generic kernel image 2026-02-20T22:21:33.319 INFO:teuthology.orchestra.run.trial107.stdout: available. 2026-02-20T22:21:33.319 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2026-02-20T22:21:33.320 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2026-02-20T22:21:33.320 DEBUG:teuthology.orchestra.run.trial107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-20T22:21:33.365 INFO:teuthology.orchestra.run.trial032.stdout:ttyS1 2026-02-20T22:21:33.369 INFO:teuthology.orchestra.run.trial107.stdout:ttyS1 2026-02-20T22:21:33.391 DEBUG:teuthology.parallel:result is None 2026-02-20T22:21:33.403 DEBUG:teuthology.parallel:result is None 2026-02-20T22:21:33.403 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-20T22:21:33.418 INFO:teuthology.task.internal:Creating test directory... 2026-02-20T22:21:33.418 DEBUG:teuthology.orchestra.run.trial032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T22:21:33.433 DEBUG:teuthology.orchestra.run.trial107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-20T22:21:33.448 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-20T22:21:33.458 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-20T22:21:33.471 INFO:teuthology.task.internal:Creating archive directory... 2026-02-20T22:21:33.471 DEBUG:teuthology.orchestra.run.trial032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T22:21:33.477 DEBUG:teuthology.orchestra.run.trial107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-20T22:21:33.502 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-20T22:21:33.518 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-20T22:21:33.518 DEBUG:teuthology.orchestra.run.trial032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T22:21:33.526 DEBUG:teuthology.orchestra.run.trial107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-20T22:21:33.533 INFO:teuthology.orchestra.run.trial032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T22:21:33.539 INFO:teuthology.orchestra.run.trial032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T22:21:33.545 INFO:teuthology.orchestra.run.trial107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T22:21:33.552 INFO:teuthology.orchestra.run.trial107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-20T22:21:33.553 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-20T22:21:33.565 INFO:teuthology.task.internal:Configuring sudo... 2026-02-20T22:21:33.565 DEBUG:teuthology.orchestra.run.trial032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T22:21:33.586 DEBUG:teuthology.orchestra.run.trial107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-20T22:21:33.605 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-20T22:21:33.621 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-20T22:21:33.621 DEBUG:teuthology.orchestra.run.trial032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T22:21:33.642 DEBUG:teuthology.orchestra.run.trial107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-20T22:21:33.656 DEBUG:teuthology.orchestra.run.trial032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T22:21:33.688 DEBUG:teuthology.orchestra.run.trial032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T22:21:33.733 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:21:33.733 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T22:21:33.782 DEBUG:teuthology.orchestra.run.trial107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T22:21:33.786 DEBUG:teuthology.orchestra.run.trial107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T22:21:33.833 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:21:33.833 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-20T22:21:33.882 DEBUG:teuthology.orchestra.run.trial032:> sudo service rsyslog restart 2026-02-20T22:21:33.884 DEBUG:teuthology.orchestra.run.trial107:> sudo service rsyslog restart 2026-02-20T22:21:33.946 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-20T22:21:33.957 INFO:teuthology.task.internal:Starting timer... 2026-02-20T22:21:33.957 INFO:teuthology.run_tasks:Running task pcp... 2026-02-20T22:21:33.979 INFO:teuthology.run_tasks:Running task selinux... 2026-02-20T22:21:34.001 DEBUG:teuthology.task.selinux:Excluding trial032: OS 'ubuntu' does not support SELinux 2026-02-20T22:21:34.002 DEBUG:teuthology.task.selinux:Excluding trial107: OS 'ubuntu' does not support SELinux 2026-02-20T22:21:34.002 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-20T22:21:34.002 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-20T22:21:34.002 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-20T22:21:34.002 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-20T22:21:34.018 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-20T22:21:34.030 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-20T22:21:34.031 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial032.front.sepia.ceph.com,trial107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-20T22:23:59.904 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial032.front.sepia.ceph.com'), Remote(name='ubuntu@trial107.front.sepia.ceph.com')] 2026-02-20T22:23:59.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial032.front.sepia.ceph.com' 2026-02-20T22:23:59.908 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T22:24:00.017 DEBUG:teuthology.orchestra.run.trial032:> true 2026-02-20T22:24:00.265 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial032.front.sepia.ceph.com' 2026-02-20T22:24:00.266 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial107.front.sepia.ceph.com' 2026-02-20T22:24:00.267 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-20T22:24:00.374 DEBUG:teuthology.orchestra.run.trial107:> true 2026-02-20T22:24:00.620 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial107.front.sepia.ceph.com' 2026-02-20T22:24:00.620 INFO:teuthology.run_tasks:Running task clock... 2026-02-20T22:24:00.644 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-20T22:24:00.644 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T22:24:00.644 DEBUG:teuthology.orchestra.run.trial032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T22:24:00.645 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-20T22:24:00.645 DEBUG:teuthology.orchestra.run.trial107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Command line: ntpd -gq 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: ---------------------------------------------------- 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: ntp-4 is maintained by Network Time Foundation, 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: corporation. Support and training for ntp-4 are 2026-02-20T22:24:00.662 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: available at https://www.nwtime.org/support 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: ---------------------------------------------------- 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: proto: precision = 0.020 usec (-25) 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: basedate set to 2022-02-04 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: gps base set to 2022-02-06 (week 2196) 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: restrict ::: KOD does nothing without LIMITED. 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen and drop on 0 v6wildcard [::]:123 2026-02-20T22:24:00.663 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-20T22:24:00.664 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen normally on 2 lo 127.0.0.1:123 2026-02-20T22:24:00.664 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen normally on 3 enp1s0f0 10.20.193.32:123 2026-02-20T22:24:00.664 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen normally on 4 lo [::1]:123 2026-02-20T22:24:00.664 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:622e%2]:123 2026-02-20T22:24:00.664 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:00 ntpd[9419]: Listening on routing socket on fd #22 for interface updates 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Command line: ntpd -gq 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: ---------------------------------------------------- 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: ntp-4 is maintained by Network Time Foundation, 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: corporation. Support and training for ntp-4 are 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: available at https://www.nwtime.org/support 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: ---------------------------------------------------- 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: proto: precision = 0.020 usec (-25) 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: basedate set to 2022-02-04 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: gps base set to 2022-02-06 (week 2196) 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-20T22:24:00.679 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: restrict ::: KOD does nothing without LIMITED. 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen and drop on 0 v6wildcard [::]:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen normally on 2 lo 127.0.0.1:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen normally on 3 enp1s0f0 10.20.193.107:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen normally on 4 lo [::1]:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5e18%2]:123 2026-02-20T22:24:00.680 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:01 ntpd[9414]: Listening on routing socket on fd #22 for interface updates 2026-02-20T22:24:01.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:01.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:01.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:01.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:01.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:01.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:01.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:01.735 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:01 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:01.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:01.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:01.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:01.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:02 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:02.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:02.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:02.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:02.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:02.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:02.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:02 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.684 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.684 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.684 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.684 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.700 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.721 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.721 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:03 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.750 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.750 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.750 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:03.756 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:03.756 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:03.756 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:03.757 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:04 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:04.662 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:04.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:04.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:04.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:04.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:04.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:04.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:04.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:04 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:04.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:04.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:04.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:04.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:05 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.682 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.683 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.683 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.683 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:05 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:05.756 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:05.756 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:05.757 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:05.757 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:06 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:06.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:06.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:06.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:06.679 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:06.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:06.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:06.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:06.688 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:06 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:06.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:06.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:06.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:06.703 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.663 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.664 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.680 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.682 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.682 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.682 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:07.683 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:07.699 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.719 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:07.720 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.733 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:07.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.734 INFO:teuthology.orchestra.run.trial032.stdout:20 Feb 22:24:07 ntpd[9419]: ntpd: time set -0.193299 s 2026-02-20T22:24:07.734 INFO:teuthology.orchestra.run.trial032.stdout:ntpd: time set -0.193299s 2026-02-20T22:24:07.734 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-20T22:24:07.735 INFO:teuthology.orchestra.run.trial032.stderr:20 Feb 22:24:07 ntpd[9419]: can't open /var/log/ntpstats/loopstats.20260220: Permission denied 2026-02-20T22:24:07.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-20T22:24:07.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/rawstats.20260220: Permission denied 2026-02-20T22:24:07.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-20T22:24:07.749 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:08 ntpd[9414]: can't open /var/log/ntpstats/peerstats.20260220: Permission denied 2026-02-20T22:24:07.750 INFO:teuthology.orchestra.run.trial107.stdout:20 Feb 22:24:07 ntpd[9414]: ntpd: time set -0.795441 s 2026-02-20T22:24:07.750 INFO:teuthology.orchestra.run.trial107.stdout:ntpd: time set -0.795441s 2026-02-20T22:24:07.750 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-20T22:24:07.750 INFO:teuthology.orchestra.run.trial107.stderr:20 Feb 22:24:07 ntpd[9414]: can't open /var/log/ntpstats/loopstats.20260220: Permission denied 2026-02-20T22:24:07.757 INFO:teuthology.orchestra.run.trial032.stdout:No association ID's returned 2026-02-20T22:24:07.789 INFO:teuthology.orchestra.run.trial107.stdout: remote refid st t when poll reach delay offset jitter 2026-02-20T22:24:07.789 INFO:teuthology.orchestra.run.trial107.stdout:============================================================================== 2026-02-20T22:24:07.790 INFO:teuthology.orchestra.run.trial107.stdout: 23.186.168.132 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T22:24:07.790 INFO:teuthology.orchestra.run.trial107.stdout: ntp.shastacoe.n .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T22:24:07.790 INFO:teuthology.orchestra.run.trial107.stdout: 104.233.211.205 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-20T22:24:07.791 INFO:teuthology.run_tasks:Running task install... 2026-02-20T22:24:07.805 DEBUG:teuthology.task.install:project ceph 2026-02-20T22:24:07.805 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}} 2026-02-20T22:24:07.806 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'} 2026-02-20T22:24:07.806 INFO:teuthology.task.install:Using flavor: default 2026-02-20T22:24:07.814 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-20T22:24:07.814 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2026-02-20T22:24:07.815 DEBUG:teuthology.orchestra.run.trial032:> sudo apt-key list | grep Ceph 2026-02-20T22:24:07.816 DEBUG:teuthology.orchestra.run.trial107:> sudo apt-key list | grep Ceph 2026-02-20T22:24:07.866 INFO:teuthology.orchestra.run.trial032.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-20T22:24:07.879 INFO:teuthology.orchestra.run.trial107.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-20T22:24:07.896 INFO:teuthology.orchestra.run.trial032.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-20T22:24:07.896 INFO:teuthology.orchestra.run.trial032.stdout:uid [ unknown] Ceph.com (release key) 2026-02-20T22:24:07.898 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-20T22:24:07.898 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:24:07.909 INFO:teuthology.orchestra.run.trial107.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-20T22:24:07.909 INFO:teuthology.orchestra.run.trial107.stdout:uid [ unknown] Ceph.com (release key) 2026-02-20T22:24:07.910 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2026-02-20T22:24:07.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:24:08.093 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default/ 2026-02-20T22:24:08.093 INFO:teuthology.task.install.deb:Package version is 20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:08.102 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default/ 2026-02-20T22:24:08.102 INFO:teuthology.task.install.deb:Package version is 20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:08.244 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:08.244 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-20T22:24:08.253 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:08.253 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-20T22:24:08.254 DEBUG:teuthology.orchestra.run.trial032:> sudo apt-get update 2026-02-20T22:24:08.263 DEBUG:teuthology.orchestra.run.trial107:> sudo apt-get update 2026-02-20T22:24:08.411 INFO:teuthology.orchestra.run.trial032.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy InRelease 2026-02-20T22:24:08.411 INFO:teuthology.orchestra.run.trial107.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy InRelease 2026-02-20T22:24:08.422 INFO:teuthology.orchestra.run.trial032.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-20T22:24:08.423 INFO:teuthology.orchestra.run.trial107.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-20T22:24:08.434 INFO:teuthology.orchestra.run.trial032.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T22:24:08.435 INFO:teuthology.orchestra.run.trial107.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-20T22:24:08.435 INFO:teuthology.orchestra.run.trial032.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-20T22:24:08.438 INFO:teuthology.orchestra.run.trial032.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-20T22:24:08.445 INFO:teuthology.orchestra.run.trial107.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-20T22:24:08.446 INFO:teuthology.orchestra.run.trial107.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.9 kB] 2026-02-20T22:24:08.446 INFO:teuthology.orchestra.run.trial032.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.9 kB] 2026-02-20T22:24:08.447 INFO:teuthology.orchestra.run.trial032.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-20T22:24:08.448 INFO:teuthology.orchestra.run.trial107.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-20T22:24:08.456 INFO:teuthology.orchestra.run.trial032.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-20T22:24:08.462 INFO:teuthology.orchestra.run.trial107.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-20T22:24:08.467 INFO:teuthology.orchestra.run.trial107.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-20T22:24:08.753 INFO:teuthology.orchestra.run.trial032.stdout:Fetched 26.5 kB in 0s (78.1 kB/s) 2026-02-20T22:24:08.777 INFO:teuthology.orchestra.run.trial107.stdout:Fetched 26.5 kB in 0s (74.8 kB/s) 2026-02-20T22:24:09.268 INFO:teuthology.orchestra.run.trial032.stdout:Reading package lists... 2026-02-20T22:24:09.281 DEBUG:teuthology.orchestra.run.trial032:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-677-gf7870454-1jammy cephadm=20.2.0-677-gf7870454-1jammy ceph-mds=20.2.0-677-gf7870454-1jammy ceph-mgr=20.2.0-677-gf7870454-1jammy ceph-common=20.2.0-677-gf7870454-1jammy ceph-fuse=20.2.0-677-gf7870454-1jammy ceph-test=20.2.0-677-gf7870454-1jammy ceph-volume=20.2.0-677-gf7870454-1jammy radosgw=20.2.0-677-gf7870454-1jammy python3-rados=20.2.0-677-gf7870454-1jammy python3-rgw=20.2.0-677-gf7870454-1jammy python3-cephfs=20.2.0-677-gf7870454-1jammy python3-rbd=20.2.0-677-gf7870454-1jammy libcephfs2=20.2.0-677-gf7870454-1jammy libcephfs-dev=20.2.0-677-gf7870454-1jammy librados2=20.2.0-677-gf7870454-1jammy librbd1=20.2.0-677-gf7870454-1jammy rbd-fuse=20.2.0-677-gf7870454-1jammy rbd-mirror=20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:09.294 INFO:teuthology.orchestra.run.trial107.stdout:Reading package lists... 2026-02-20T22:24:09.308 DEBUG:teuthology.orchestra.run.trial107:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-677-gf7870454-1jammy cephadm=20.2.0-677-gf7870454-1jammy ceph-mds=20.2.0-677-gf7870454-1jammy ceph-mgr=20.2.0-677-gf7870454-1jammy ceph-common=20.2.0-677-gf7870454-1jammy ceph-fuse=20.2.0-677-gf7870454-1jammy ceph-test=20.2.0-677-gf7870454-1jammy ceph-volume=20.2.0-677-gf7870454-1jammy radosgw=20.2.0-677-gf7870454-1jammy python3-rados=20.2.0-677-gf7870454-1jammy python3-rgw=20.2.0-677-gf7870454-1jammy python3-cephfs=20.2.0-677-gf7870454-1jammy python3-rbd=20.2.0-677-gf7870454-1jammy libcephfs2=20.2.0-677-gf7870454-1jammy libcephfs-dev=20.2.0-677-gf7870454-1jammy librados2=20.2.0-677-gf7870454-1jammy librbd1=20.2.0-677-gf7870454-1jammy rbd-fuse=20.2.0-677-gf7870454-1jammy rbd-mirror=20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:09.314 INFO:teuthology.orchestra.run.trial032.stdout:Reading package lists... 2026-02-20T22:24:09.342 INFO:teuthology.orchestra.run.trial107.stdout:Reading package lists... 2026-02-20T22:24:09.420 INFO:teuthology.orchestra.run.trial032.stdout:Building dependency tree... 2026-02-20T22:24:09.420 INFO:teuthology.orchestra.run.trial032.stdout:Reading state information... 2026-02-20T22:24:09.451 INFO:teuthology.orchestra.run.trial107.stdout:Building dependency tree... 2026-02-20T22:24:09.452 INFO:teuthology.orchestra.run.trial107.stdout:Reading state information... 2026-02-20T22:24:09.510 INFO:teuthology.orchestra.run.trial032.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: sg3-utils-udev 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout:The following additional packages will be installed: 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-20T22:24:09.511 INFO:teuthology.orchestra.run.trial032.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout: socat xmlstarlet 2026-02-20T22:24:09.512 INFO:teuthology.orchestra.run.trial032.stdout:Suggested packages: 2026-02-20T22:24:09.513 INFO:teuthology.orchestra.run.trial032.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-20T22:24:09.513 INFO:teuthology.orchestra.run.trial032.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-20T22:24:09.513 INFO:teuthology.orchestra.run.trial032.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-20T22:24:09.513 INFO:teuthology.orchestra.run.trial032.stdout:Recommended packages: 2026-02-20T22:24:09.513 INFO:teuthology.orchestra.run.trial032.stdout: btrfs-tools 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout:The following packages were automatically installed and are no longer required: 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout: sg3-utils-udev 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-20T22:24:09.543 INFO:teuthology.orchestra.run.trial107.stdout:The following additional packages will be installed: 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-20T22:24:09.544 INFO:teuthology.orchestra.run.trial107.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: socat xmlstarlet 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout:Suggested packages: 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout:Recommended packages: 2026-02-20T22:24:09.545 INFO:teuthology.orchestra.run.trial107.stdout: btrfs-tools 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout:The following NEW packages will be installed: 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-20T22:24:09.547 INFO:teuthology.orchestra.run.trial032.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-20T22:24:09.548 INFO:teuthology.orchestra.run.trial032.stdout: rbd-mirror socat xmlstarlet 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout:The following NEW packages will be installed: 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-20T22:24:09.577 INFO:teuthology.orchestra.run.trial107.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-20T22:24:09.578 INFO:teuthology.orchestra.run.trial107.stdout: rbd-mirror socat xmlstarlet 2026-02-20T22:24:09.647 INFO:teuthology.orchestra.run.trial032.stdout:0 upgraded, 87 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T22:24:09.647 INFO:teuthology.orchestra.run.trial032.stdout:Need to get 283 MB of archives. 2026-02-20T22:24:09.647 INFO:teuthology.orchestra.run.trial032.stdout:After this operation, 1130 MB of additional disk space will be used. 2026-02-20T22:24:09.647 INFO:teuthology.orchestra.run.trial032.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-20T22:24:09.654 INFO:teuthology.orchestra.run.trial107.stdout:0 upgraded, 87 newly installed, 0 to remove and 13 not upgraded. 2026-02-20T22:24:09.654 INFO:teuthology.orchestra.run.trial107.stdout:Need to get 283 MB of archives. 2026-02-20T22:24:09.654 INFO:teuthology.orchestra.run.trial107.stdout:After this operation, 1130 MB of additional disk space will be used. 2026-02-20T22:24:09.654 INFO:teuthology.orchestra.run.trial107.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-677-gf7870454-1jammy [3562 kB] 2026-02-20T22:24:09.691 INFO:teuthology.orchestra.run.trial032.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-20T22:24:09.693 INFO:teuthology.orchestra.run.trial032.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-20T22:24:09.703 INFO:teuthology.orchestra.run.trial032.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-20T22:24:09.731 INFO:teuthology.orchestra.run.trial032.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-20T22:24:09.732 INFO:teuthology.orchestra.run.trial032.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-20T22:24:09.738 INFO:teuthology.orchestra.run.trial032.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-20T22:24:09.739 INFO:teuthology.orchestra.run.trial032.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-20T22:24:09.740 INFO:teuthology.orchestra.run.trial032.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-20T22:24:09.740 INFO:teuthology.orchestra.run.trial032.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-20T22:24:09.741 INFO:teuthology.orchestra.run.trial032.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-20T22:24:09.745 INFO:teuthology.orchestra.run.trial032.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-20T22:24:09.746 INFO:teuthology.orchestra.run.trial032.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-20T22:24:09.746 INFO:teuthology.orchestra.run.trial032.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-20T22:24:09.751 INFO:teuthology.orchestra.run.trial032.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-20T22:24:09.752 INFO:teuthology.orchestra.run.trial032.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-20T22:24:09.752 INFO:teuthology.orchestra.run.trial032.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-20T22:24:09.752 INFO:teuthology.orchestra.run.trial032.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-20T22:24:09.761 INFO:teuthology.orchestra.run.trial032.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-20T22:24:09.761 INFO:teuthology.orchestra.run.trial032.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-20T22:24:09.761 INFO:teuthology.orchestra.run.trial032.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-20T22:24:09.763 INFO:teuthology.orchestra.run.trial032.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-20T22:24:09.763 INFO:teuthology.orchestra.run.trial032.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-20T22:24:09.764 INFO:teuthology.orchestra.run.trial032.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-20T22:24:09.765 INFO:teuthology.orchestra.run.trial032.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-677-gf7870454-1jammy [3562 kB] 2026-02-20T22:24:09.768 INFO:teuthology.orchestra.run.trial107.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-677-gf7870454-1jammy [2867 kB] 2026-02-20T22:24:09.770 INFO:teuthology.orchestra.run.trial032.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-20T22:24:09.771 INFO:teuthology.orchestra.run.trial032.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-20T22:24:09.771 INFO:teuthology.orchestra.run.trial032.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-20T22:24:09.785 INFO:teuthology.orchestra.run.trial032.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-20T22:24:09.787 INFO:teuthology.orchestra.run.trial032.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-20T22:24:09.787 INFO:teuthology.orchestra.run.trial032.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-20T22:24:09.797 INFO:teuthology.orchestra.run.trial032.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-20T22:24:09.797 INFO:teuthology.orchestra.run.trial032.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-20T22:24:09.797 INFO:teuthology.orchestra.run.trial032.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-20T22:24:09.798 INFO:teuthology.orchestra.run.trial032.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-20T22:24:09.798 INFO:teuthology.orchestra.run.trial032.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-20T22:24:09.798 INFO:teuthology.orchestra.run.trial032.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-20T22:24:09.799 INFO:teuthology.orchestra.run.trial032.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-20T22:24:09.800 INFO:teuthology.orchestra.run.trial032.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-20T22:24:09.801 INFO:teuthology.orchestra.run.trial032.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-20T22:24:09.808 INFO:teuthology.orchestra.run.trial032.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-20T22:24:09.810 INFO:teuthology.orchestra.run.trial032.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-20T22:24:09.811 INFO:teuthology.orchestra.run.trial032.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-20T22:24:09.813 INFO:teuthology.orchestra.run.trial032.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-20T22:24:09.814 INFO:teuthology.orchestra.run.trial032.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-20T22:24:09.818 INFO:teuthology.orchestra.run.trial032.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-20T22:24:09.818 INFO:teuthology.orchestra.run.trial032.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-20T22:24:09.819 INFO:teuthology.orchestra.run.trial032.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-20T22:24:09.819 INFO:teuthology.orchestra.run.trial032.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-20T22:24:09.822 INFO:teuthology.orchestra.run.trial032.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-20T22:24:09.825 INFO:teuthology.orchestra.run.trial107.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-677-gf7870454-1jammy [833 kB] 2026-02-20T22:24:09.827 INFO:teuthology.orchestra.run.trial032.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-20T22:24:09.828 INFO:teuthology.orchestra.run.trial032.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-20T22:24:09.829 INFO:teuthology.orchestra.run.trial032.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-20T22:24:09.829 INFO:teuthology.orchestra.run.trial032.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-20T22:24:09.839 INFO:teuthology.orchestra.run.trial107.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-677-gf7870454-1jammy [364 kB] 2026-02-20T22:24:09.846 INFO:teuthology.orchestra.run.trial107.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-677-gf7870454-1jammy [32.8 kB] 2026-02-20T22:24:09.848 INFO:teuthology.orchestra.run.trial107.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-677-gf7870454-1jammy [184 kB] 2026-02-20T22:24:09.854 INFO:teuthology.orchestra.run.trial107.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-677-gf7870454-1jammy [83.9 kB] 2026-02-20T22:24:09.856 INFO:teuthology.orchestra.run.trial107.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-677-gf7870454-1jammy [341 kB] 2026-02-20T22:24:09.863 INFO:teuthology.orchestra.run.trial107.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-677-gf7870454-1jammy [8698 kB] 2026-02-20T22:24:09.926 INFO:teuthology.orchestra.run.trial107.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-20T22:24:10.044 INFO:teuthology.orchestra.run.trial107.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-677-gf7870454-1jammy [112 kB] 2026-02-20T22:24:10.045 INFO:teuthology.orchestra.run.trial107.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-677-gf7870454-1jammy [261 kB] 2026-02-20T22:24:10.049 INFO:teuthology.orchestra.run.trial107.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-677-gf7870454-1jammy [29.3 MB] 2026-02-20T22:24:10.262 INFO:teuthology.orchestra.run.trial107.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-20T22:24:10.272 INFO:teuthology.orchestra.run.trial107.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-20T22:24:10.342 INFO:teuthology.orchestra.run.trial107.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-20T22:24:10.543 INFO:teuthology.orchestra.run.trial107.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-20T22:24:10.553 INFO:teuthology.orchestra.run.trial107.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-20T22:24:10.579 INFO:teuthology.orchestra.run.trial107.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-20T22:24:10.588 INFO:teuthology.orchestra.run.trial107.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-20T22:24:10.589 INFO:teuthology.orchestra.run.trial107.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-20T22:24:10.590 INFO:teuthology.orchestra.run.trial107.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-20T22:24:10.591 INFO:teuthology.orchestra.run.trial107.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-20T22:24:10.608 INFO:teuthology.orchestra.run.trial107.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-20T22:24:10.608 INFO:teuthology.orchestra.run.trial107.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-20T22:24:10.615 INFO:teuthology.orchestra.run.trial107.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-20T22:24:10.642 INFO:teuthology.orchestra.run.trial107.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-677-gf7870454-1jammy [5416 kB] 2026-02-20T22:24:10.681 INFO:teuthology.orchestra.run.trial107.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-20T22:24:10.681 INFO:teuthology.orchestra.run.trial107.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-20T22:24:10.681 INFO:teuthology.orchestra.run.trial107.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-20T22:24:10.681 INFO:teuthology.orchestra.run.trial107.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-20T22:24:10.682 INFO:teuthology.orchestra.run.trial107.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-20T22:24:10.682 INFO:teuthology.orchestra.run.trial107.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-20T22:24:10.684 INFO:teuthology.orchestra.run.trial107.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-20T22:24:10.686 INFO:teuthology.orchestra.run.trial107.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-20T22:24:10.687 INFO:teuthology.orchestra.run.trial107.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-20T22:24:10.748 INFO:teuthology.orchestra.run.trial107.stdout:Get:37 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-677-gf7870454-1jammy [246 kB] 2026-02-20T22:24:10.752 INFO:teuthology.orchestra.run.trial107.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-677-gf7870454-1jammy [124 kB] 2026-02-20T22:24:10.754 INFO:teuthology.orchestra.run.trial107.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-677-gf7870454-1jammy [906 kB] 2026-02-20T22:24:10.758 INFO:teuthology.orchestra.run.trial107.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-20T22:24:10.760 INFO:teuthology.orchestra.run.trial107.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-20T22:24:10.761 INFO:teuthology.orchestra.run.trial107.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-20T22:24:10.762 INFO:teuthology.orchestra.run.trial107.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-20T22:24:10.772 INFO:teuthology.orchestra.run.trial107.stdout:Get:44 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-677-gf7870454-1jammy [6401 kB] 2026-02-20T22:24:10.786 INFO:teuthology.orchestra.run.trial107.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-20T22:24:10.789 INFO:teuthology.orchestra.run.trial107.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-20T22:24:10.789 INFO:teuthology.orchestra.run.trial107.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-20T22:24:10.846 INFO:teuthology.orchestra.run.trial107.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-20T22:24:10.847 INFO:teuthology.orchestra.run.trial107.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-20T22:24:10.847 INFO:teuthology.orchestra.run.trial107.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-20T22:24:10.901 INFO:teuthology.orchestra.run.trial107.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-677-gf7870454-1jammy [21.7 MB] 2026-02-20T22:24:10.902 INFO:teuthology.orchestra.run.trial107.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-20T22:24:10.902 INFO:teuthology.orchestra.run.trial107.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-20T22:24:10.903 INFO:teuthology.orchestra.run.trial107.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-20T22:24:10.905 INFO:teuthology.orchestra.run.trial107.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-20T22:24:10.906 INFO:teuthology.orchestra.run.trial107.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-20T22:24:10.908 INFO:teuthology.orchestra.run.trial107.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-20T22:24:10.974 INFO:teuthology.orchestra.run.trial107.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-20T22:24:10.977 INFO:teuthology.orchestra.run.trial107.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-20T22:24:10.979 INFO:teuthology.orchestra.run.trial107.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-20T22:24:10.982 INFO:teuthology.orchestra.run.trial107.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-20T22:24:11.046 INFO:teuthology.orchestra.run.trial107.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-20T22:24:11.046 INFO:teuthology.orchestra.run.trial107.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-20T22:24:11.046 INFO:teuthology.orchestra.run.trial107.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-20T22:24:11.047 INFO:teuthology.orchestra.run.trial107.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-20T22:24:11.048 INFO:teuthology.orchestra.run.trial107.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-20T22:24:11.054 INFO:teuthology.orchestra.run.trial107.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-20T22:24:11.118 INFO:teuthology.orchestra.run.trial107.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-20T22:24:11.119 INFO:teuthology.orchestra.run.trial107.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-20T22:24:11.120 INFO:teuthology.orchestra.run.trial107.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-20T22:24:11.120 INFO:teuthology.orchestra.run.trial107.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-20T22:24:11.220 INFO:teuthology.orchestra.run.trial032.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-677-gf7870454-1jammy [2867 kB] 2026-02-20T22:24:11.343 INFO:teuthology.orchestra.run.trial107.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-677-gf7870454-1jammy [14.1 kB] 2026-02-20T22:24:11.343 INFO:teuthology.orchestra.run.trial107.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-677-gf7870454-1jammy [957 kB] 2026-02-20T22:24:11.363 INFO:teuthology.orchestra.run.trial107.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-677-gf7870454-1jammy [2345 kB] 2026-02-20T22:24:11.408 INFO:teuthology.orchestra.run.trial107.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-677-gf7870454-1jammy [1049 kB] 2026-02-20T22:24:11.432 INFO:teuthology.orchestra.run.trial107.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-677-gf7870454-1jammy [179 kB] 2026-02-20T22:24:11.436 INFO:teuthology.orchestra.run.trial107.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-677-gf7870454-1jammy [45.5 MB] 2026-02-20T22:24:11.789 INFO:teuthology.orchestra.run.trial032.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-677-gf7870454-1jammy [833 kB] 2026-02-20T22:24:11.919 INFO:teuthology.orchestra.run.trial032.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-677-gf7870454-1jammy [364 kB] 2026-02-20T22:24:11.982 INFO:teuthology.orchestra.run.trial032.stdout:Get:58 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-677-gf7870454-1jammy [32.8 kB] 2026-02-20T22:24:11.983 INFO:teuthology.orchestra.run.trial032.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-677-gf7870454-1jammy [184 kB] 2026-02-20T22:24:11.986 INFO:teuthology.orchestra.run.trial032.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-677-gf7870454-1jammy [83.9 kB] 2026-02-20T22:24:11.988 INFO:teuthology.orchestra.run.trial032.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-677-gf7870454-1jammy [341 kB] 2026-02-20T22:24:12.051 INFO:teuthology.orchestra.run.trial032.stdout:Get:62 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-677-gf7870454-1jammy [8698 kB] 2026-02-20T22:24:12.423 INFO:teuthology.orchestra.run.trial107.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-677-gf7870454-1jammy [8625 kB] 2026-02-20T22:24:12.632 INFO:teuthology.orchestra.run.trial107.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-677-gf7870454-1jammy [14.2 kB] 2026-02-20T22:24:12.632 INFO:teuthology.orchestra.run.trial107.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-677-gf7870454-1jammy [99.5 MB] 2026-02-20T22:24:12.851 INFO:teuthology.orchestra.run.trial032.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-677-gf7870454-1jammy [112 kB] 2026-02-20T22:24:12.851 INFO:teuthology.orchestra.run.trial032.stdout:Get:64 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-677-gf7870454-1jammy [261 kB] 2026-02-20T22:24:12.883 INFO:teuthology.orchestra.run.trial032.stdout:Get:65 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-677-gf7870454-1jammy [29.3 MB] 2026-02-20T22:24:14.393 INFO:teuthology.orchestra.run.trial032.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-677-gf7870454-1jammy [5416 kB] 2026-02-20T22:24:14.625 INFO:teuthology.orchestra.run.trial032.stdout:Get:67 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-677-gf7870454-1jammy [246 kB] 2026-02-20T22:24:14.655 INFO:teuthology.orchestra.run.trial032.stdout:Get:68 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-677-gf7870454-1jammy [124 kB] 2026-02-20T22:24:14.660 INFO:teuthology.orchestra.run.trial032.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-677-gf7870454-1jammy [906 kB] 2026-02-20T22:24:14.681 INFO:teuthology.orchestra.run.trial032.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-677-gf7870454-1jammy [6401 kB] 2026-02-20T22:24:15.001 INFO:teuthology.orchestra.run.trial032.stdout:Get:71 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-677-gf7870454-1jammy [21.7 MB] 2026-02-20T22:24:15.478 INFO:teuthology.orchestra.run.trial107.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-677-gf7870454-1jammy [135 kB] 2026-02-20T22:24:15.478 INFO:teuthology.orchestra.run.trial107.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-677-gf7870454-1jammy [43.3 kB] 2026-02-20T22:24:15.479 INFO:teuthology.orchestra.run.trial107.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-677-gf7870454-1jammy [30.7 kB] 2026-02-20T22:24:15.479 INFO:teuthology.orchestra.run.trial107.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-677-gf7870454-1jammy [41.5 kB] 2026-02-20T22:24:15.479 INFO:teuthology.orchestra.run.trial107.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-677-gf7870454-1jammy [25.1 MB] 2026-02-20T22:24:15.992 INFO:teuthology.orchestra.run.trial032.stdout:Get:72 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-677-gf7870454-1jammy [14.1 kB] 2026-02-20T22:24:15.993 INFO:teuthology.orchestra.run.trial032.stdout:Get:73 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-677-gf7870454-1jammy [957 kB] 2026-02-20T22:24:16.039 INFO:teuthology.orchestra.run.trial032.stdout:Get:74 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-677-gf7870454-1jammy [2345 kB] 2026-02-20T22:24:16.147 INFO:teuthology.orchestra.run.trial032.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-677-gf7870454-1jammy [1049 kB] 2026-02-20T22:24:16.176 INFO:teuthology.orchestra.run.trial032.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-677-gf7870454-1jammy [179 kB] 2026-02-20T22:24:16.187 INFO:teuthology.orchestra.run.trial032.stdout:Get:77 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-677-gf7870454-1jammy [45.5 MB] 2026-02-20T22:24:16.225 INFO:teuthology.orchestra.run.trial107.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-677-gf7870454-1jammy [97.9 kB] 2026-02-20T22:24:16.226 INFO:teuthology.orchestra.run.trial107.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 20.2.0-677-gf7870454-1jammy [2883 kB] 2026-02-20T22:24:16.598 INFO:teuthology.orchestra.run.trial107.stdout:Fetched 283 MB in 7s (42.0 MB/s) 2026-02-20T22:24:16.615 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-20T22:24:16.771 INFO:teuthology.orchestra.run.trial107.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-20T22:24:16.773 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-20T22:24:16.775 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T22:24:16.795 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-20T22:24:16.808 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-20T22:24:16.808 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T22:24:16.823 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-20T22:24:16.835 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-20T22:24:16.836 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T22:24:16.855 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-20T22:24:16.863 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:16.868 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:16.897 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-20T22:24:16.905 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:16.906 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:16.923 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-20T22:24:16.931 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:16.931 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:16.951 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-20T22:24:16.959 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-20T22:24:16.959 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T22:24:16.975 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package librados2. 2026-02-20T22:24:16.983 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../07-librados2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:16.983 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.025 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libnbd0. 2026-02-20T22:24:17.033 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-20T22:24:17.033 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-20T22:24:17.047 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package librbd1. 2026-02-20T22:24:17.055 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../09-librbd1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.055 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.093 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libcephfs2. 2026-02-20T22:24:17.101 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.102 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.121 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-rados. 2026-02-20T22:24:17.129 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../11-python3-rados_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.130 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.147 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-20T22:24:17.155 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:17.155 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.169 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-cephfs. 2026-02-20T22:24:17.177 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.177 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.193 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-20T22:24:17.201 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:17.202 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.220 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-20T22:24:17.228 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-20T22:24:17.229 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T22:24:17.244 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-prettytable. 2026-02-20T22:24:17.252 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-20T22:24:17.252 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-20T22:24:17.266 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-rbd. 2026-02-20T22:24:17.274 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.275 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.291 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-20T22:24:17.299 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-20T22:24:17.300 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T22:24:17.318 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package librgw2. 2026-02-20T22:24:17.326 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../19-librgw2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.326 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.423 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-rgw. 2026-02-20T22:24:17.431 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.431 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.447 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-20T22:24:17.455 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-20T22:24:17.455 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T22:24:17.469 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libradosstriper1. 2026-02-20T22:24:17.477 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.478 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.484 INFO:teuthology.orchestra.run.trial032.stdout:Get:78 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-677-gf7870454-1jammy [8625 kB] 2026-02-20T22:24:17.494 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-common. 2026-02-20T22:24:17.502 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../23-ceph-common_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.502 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.651 INFO:teuthology.orchestra.run.trial032.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-677-gf7870454-1jammy [14.2 kB] 2026-02-20T22:24:17.652 INFO:teuthology.orchestra.run.trial032.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-677-gf7870454-1jammy [99.5 MB] 2026-02-20T22:24:17.755 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-base. 2026-02-20T22:24:17.763 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../24-ceph-base_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:17.768 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:17.834 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-20T22:24:17.842 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-20T22:24:17.843 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-20T22:24:17.857 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-cheroot. 2026-02-20T22:24:17.866 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-20T22:24:17.866 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T22:24:17.883 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-20T22:24:17.891 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-20T22:24:17.892 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-20T22:24:17.906 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-20T22:24:17.914 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-20T22:24:17.915 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-20T22:24:17.928 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-20T22:24:17.936 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-20T22:24:17.936 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-20T22:24:17.951 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-tempora. 2026-02-20T22:24:17.959 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-20T22:24:17.959 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-20T22:24:17.973 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-portend. 2026-02-20T22:24:17.981 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-20T22:24:17.982 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-20T22:24:17.995 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-20T22:24:18.003 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-20T22:24:18.004 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-20T22:24:18.019 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-20T22:24:18.027 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-20T22:24:18.027 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-20T22:24:18.050 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-natsort. 2026-02-20T22:24:18.058 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-20T22:24:18.059 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-20T22:24:18.075 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-20T22:24:18.083 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:18.083 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.109 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-20T22:24:18.117 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.118 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.132 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr. 2026-02-20T22:24:18.140 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.141 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.161 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mon. 2026-02-20T22:24:18.169 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.169 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.229 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-20T22:24:18.237 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-20T22:24:18.237 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T22:24:18.253 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-osd. 2026-02-20T22:24:18.261 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.261 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.404 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph. 2026-02-20T22:24:18.413 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../41-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.413 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.427 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-fuse. 2026-02-20T22:24:18.435 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.436 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.457 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mds. 2026-02-20T22:24:18.465 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.465 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.496 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package cephadm. 2026-02-20T22:24:18.505 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../44-cephadm_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:18.505 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.521 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-20T22:24:18.529 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T22:24:18.529 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:18.551 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-20T22:24:18.559 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:18.559 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:18.580 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-20T22:24:18.588 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-20T22:24:18.588 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-20T22:24:18.603 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-routes. 2026-02-20T22:24:18.611 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-20T22:24:18.612 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T22:24:18.631 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-20T22:24:18.638 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:18.639 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:19.059 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-20T22:24:19.068 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-20T22:24:19.069 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T22:24:19.108 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-joblib. 2026-02-20T22:24:19.116 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-20T22:24:19.117 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T22:24:19.145 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-20T22:24:19.153 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-20T22:24:19.154 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-20T22:24:19.169 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-sklearn. 2026-02-20T22:24:19.177 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-20T22:24:19.178 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T22:24:19.259 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-20T22:24:19.268 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:19.268 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:19.435 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-cachetools. 2026-02-20T22:24:19.443 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-20T22:24:19.444 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-20T22:24:19.459 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-rsa. 2026-02-20T22:24:19.467 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-20T22:24:19.467 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-20T22:24:19.486 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-google-auth. 2026-02-20T22:24:19.495 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-20T22:24:19.495 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-20T22:24:19.512 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-20T22:24:19.521 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-20T22:24:19.521 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T22:24:19.537 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-websocket. 2026-02-20T22:24:19.546 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-20T22:24:19.546 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-20T22:24:19.563 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-20T22:24:19.574 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-20T22:24:19.575 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T22:24:19.649 INFO:teuthology.orchestra.run.trial032.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-677-gf7870454-1jammy [135 kB] 2026-02-20T22:24:19.650 INFO:teuthology.orchestra.run.trial032.stdout:Get:82 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-677-gf7870454-1jammy [43.3 kB] 2026-02-20T22:24:19.650 INFO:teuthology.orchestra.run.trial032.stdout:Get:83 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-677-gf7870454-1jammy [30.7 kB] 2026-02-20T22:24:19.651 INFO:teuthology.orchestra.run.trial032.stdout:Get:84 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-677-gf7870454-1jammy [41.5 kB] 2026-02-20T22:24:19.651 INFO:teuthology.orchestra.run.trial032.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-677-gf7870454-1jammy [25.1 MB] 2026-02-20T22:24:19.668 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-20T22:24:19.677 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:19.678 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:19.692 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-20T22:24:19.700 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-20T22:24:19.700 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T22:24:19.716 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-20T22:24:19.725 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T22:24:19.725 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:19.740 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package jq. 2026-02-20T22:24:19.749 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T22:24:19.749 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:19.763 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package socat. 2026-02-20T22:24:19.771 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-20T22:24:19.772 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-20T22:24:19.791 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package xmlstarlet. 2026-02-20T22:24:19.799 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-20T22:24:19.800 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-20T22:24:19.836 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-test. 2026-02-20T22:24:19.845 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../67-ceph-test_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:19.845 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.146 INFO:teuthology.orchestra.run.trial032.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-677-gf7870454-1jammy [97.9 kB] 2026-02-20T22:24:20.147 INFO:teuthology.orchestra.run.trial032.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20260219.182737-tentacle/f7870454149f22574086dd2ff55d050585136531/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 20.2.0-677-gf7870454-1jammy [2883 kB] 2026-02-20T22:24:20.480 INFO:teuthology.orchestra.run.trial032.stdout:Fetched 283 MB in 11s (26.6 MB/s) 2026-02-20T22:24:20.499 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-20T22:24:20.658 INFO:teuthology.orchestra.run.trial032.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-20T22:24:20.660 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-20T22:24:20.662 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T22:24:20.680 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-20T22:24:20.692 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-20T22:24:20.693 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T22:24:20.707 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-20T22:24:20.720 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-20T22:24:20.720 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T22:24:20.738 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-20T22:24:20.747 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:20.751 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:20.762 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package ceph-volume. 2026-02-20T22:24:20.771 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:20.772 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.781 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-20T22:24:20.789 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:20.790 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:20.795 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-20T22:24:20.803 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.804 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.806 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-20T22:24:20.814 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-20T22:24:20.815 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:20.818 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-20T22:24:20.826 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.827 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.834 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-20T22:24:20.841 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-20T22:24:20.842 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-20T22:24:20.843 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T22:24:20.850 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.850 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.858 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package librados2. 2026-02-20T22:24:20.866 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package nvme-cli. 2026-02-20T22:24:20.866 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../07-librados2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.867 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.874 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-20T22:24:20.874 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T22:24:20.906 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-20T22:24:20.908 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libnbd0. 2026-02-20T22:24:20.915 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T22:24:20.915 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:20.916 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-20T22:24:20.917 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-20T22:24:20.930 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package librbd1. 2026-02-20T22:24:20.938 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../09-librbd1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.939 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.944 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-20T22:24:20.952 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-20T22:24:20.953 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-20T22:24:20.967 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-pluggy. 2026-02-20T22:24:20.975 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-20T22:24:20.975 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-20T22:24:20.976 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libcephfs2. 2026-02-20T22:24:20.984 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:20.984 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:20.991 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-psutil. 2026-02-20T22:24:20.999 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-20T22:24:21.000 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-20T22:24:21.003 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-rados. 2026-02-20T22:24:21.011 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../11-python3-rados_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.012 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.018 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-py. 2026-02-20T22:24:21.027 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-20T22:24:21.027 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-20T22:24:21.029 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-20T22:24:21.037 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:21.038 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.047 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-pygments. 2026-02-20T22:24:21.052 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-cephfs. 2026-02-20T22:24:21.055 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-20T22:24:21.056 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T22:24:21.060 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.060 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.076 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-20T22:24:21.084 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:21.085 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.096 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-toml. 2026-02-20T22:24:21.102 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-20T22:24:21.105 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-20T22:24:21.106 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-20T22:24:21.110 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-20T22:24:21.111 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T22:24:21.120 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-pytest. 2026-02-20T22:24:21.126 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-prettytable. 2026-02-20T22:24:21.128 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-20T22:24:21.128 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T22:24:21.134 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-20T22:24:21.135 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-20T22:24:21.148 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-rbd. 2026-02-20T22:24:21.150 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-simplejson. 2026-02-20T22:24:21.157 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.157 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.158 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-20T22:24:21.158 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-20T22:24:21.173 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-20T22:24:21.177 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package python3-webob. 2026-02-20T22:24:21.182 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-20T22:24:21.182 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T22:24:21.185 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-20T22:24:21.185 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T22:24:21.199 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package librgw2. 2026-02-20T22:24:21.207 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../19-librgw2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.208 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.208 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-20T22:24:21.216 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-20T22:24:21.217 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-20T22:24:21.285 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package radosgw. 2026-02-20T22:24:21.294 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../84-radosgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.294 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.300 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-rgw. 2026-02-20T22:24:21.309 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.309 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.325 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-20T22:24:21.332 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-20T22:24:21.333 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T22:24:21.347 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libradosstriper1. 2026-02-20T22:24:21.356 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.356 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.372 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-common. 2026-02-20T22:24:21.380 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../23-ceph-common_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.380 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.559 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package rbd-fuse. 2026-02-20T22:24:21.568 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.568 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.583 INFO:teuthology.orchestra.run.trial107.stdout:Selecting previously unselected package rbd-mirror. 2026-02-20T22:24:21.591 INFO:teuthology.orchestra.run.trial107.stdout:Preparing to unpack .../86-rbd-mirror_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.592 INFO:teuthology.orchestra.run.trial107.stdout:Unpacking rbd-mirror (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.634 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-base. 2026-02-20T22:24:21.642 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../24-ceph-base_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.645 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-20T22:24:21.647 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.710 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-20T22:24:21.711 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T22:24:21.711 INFO:teuthology.orchestra.run.trial107.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T22:24:21.719 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-20T22:24:21.719 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-20T22:24:21.734 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-cheroot. 2026-02-20T22:24:21.742 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-20T22:24:21.743 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T22:24:21.760 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-20T22:24:21.768 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-20T22:24:21.768 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-20T22:24:21.782 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T22:24:21.783 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-20T22:24:21.791 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-20T22:24:21.791 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-20T22:24:21.805 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-20T22:24:21.812 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-20T22:24:21.813 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-20T22:24:21.828 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-tempora. 2026-02-20T22:24:21.836 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-20T22:24:21.837 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-20T22:24:21.851 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-portend. 2026-02-20T22:24:21.859 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-20T22:24:21.860 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-20T22:24:21.874 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-20T22:24:21.883 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-20T22:24:21.883 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-20T22:24:21.897 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-20T22:24:21.905 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-20T22:24:21.906 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-20T22:24:21.929 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-natsort. 2026-02-20T22:24:21.937 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-20T22:24:21.938 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-20T22:24:21.953 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-20T22:24:21.962 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:21.962 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:21.988 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-20T22:24:21.996 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:21.997 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.012 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr. 2026-02-20T22:24:22.020 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.020 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.040 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mon. 2026-02-20T22:24:22.048 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.049 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.069 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-20T22:24:22.108 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-20T22:24:22.116 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-20T22:24:22.117 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T22:24:22.133 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-osd. 2026-02-20T22:24:22.141 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.141 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.286 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph. 2026-02-20T22:24:22.294 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../41-ceph_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.295 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.308 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-fuse. 2026-02-20T22:24:22.317 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.317 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.340 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mds. 2026-02-20T22:24:22.349 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.349 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.380 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package cephadm. 2026-02-20T22:24:22.389 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../44-cephadm_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:22.390 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.406 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-20T22:24:22.414 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T22:24:22.414 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:22.436 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-20T22:24:22.444 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:22.444 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.464 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-20T22:24:22.465 INFO:teuthology.orchestra.run.trial107.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-20T22:24:22.472 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-20T22:24:22.472 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-20T22:24:22.475 INFO:teuthology.orchestra.run.trial107.stdout:Setting up cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.487 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-routes. 2026-02-20T22:24:22.496 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-20T22:24:22.496 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T22:24:22.516 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-20T22:24:22.520 INFO:teuthology.orchestra.run.trial107.stdout:Adding system user cephadm....done 2026-02-20T22:24:22.524 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:22.525 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:22.529 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-20T22:24:22.602 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:22.603 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-20T22:24:22.679 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-20T22:24:22.754 INFO:teuthology.orchestra.run.trial107.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T22:24:22.755 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-20T22:24:22.851 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T22:24:22.948 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-20T22:24:22.958 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-20T22:24:22.959 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T22:24:22.974 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-20T22:24:22.997 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-joblib. 2026-02-20T22:24:23.005 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-20T22:24:23.006 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T22:24:23.032 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-20T22:24:23.041 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-20T22:24:23.041 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-20T22:24:23.049 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-20T22:24:23.056 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-sklearn. 2026-02-20T22:24:23.064 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-20T22:24:23.065 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T22:24:23.124 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:23.146 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-20T22:24:23.155 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:23.156 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:23.192 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T22:24:23.193 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-20T22:24:23.194 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T22:24:23.195 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T22:24:23.196 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-20T22:24:23.317 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-20T22:24:23.318 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-cachetools. 2026-02-20T22:24:23.327 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-20T22:24:23.327 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-20T22:24:23.341 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-rsa. 2026-02-20T22:24:23.350 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-20T22:24:23.350 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-20T22:24:23.367 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-google-auth. 2026-02-20T22:24:23.376 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-20T22:24:23.376 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-20T22:24:23.394 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-20T22:24:23.396 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:23.397 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T22:24:23.402 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-20T22:24:23.403 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T22:24:23.419 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-websocket. 2026-02-20T22:24:23.427 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-20T22:24:23.428 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-20T22:24:23.445 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-20T22:24:23.454 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-20T22:24:23.454 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T22:24:23.477 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-20T22:24:23.554 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-20T22:24:23.563 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:23.563 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:23.566 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T22:24:23.576 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-20T22:24:23.585 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-20T22:24:23.585 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T22:24:23.601 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-20T22:24:23.610 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T22:24:23.610 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:23.625 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package jq. 2026-02-20T22:24:23.634 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-20T22:24:23.634 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:23.647 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package socat. 2026-02-20T22:24:23.656 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-20T22:24:23.656 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-20T22:24:23.674 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package xmlstarlet. 2026-02-20T22:24:23.683 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-20T22:24:23.683 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-20T22:24:23.717 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-test. 2026-02-20T22:24:23.726 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../67-ceph-test_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:23.727 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:23.818 INFO:teuthology.orchestra.run.trial107.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-20T22:24:23.819 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T22:24:23.915 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:24.043 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T22:24:24.125 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-20T22:24:24.199 INFO:teuthology.orchestra.run.trial107.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-20T22:24:24.199 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:24.292 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T22:24:24.632 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package ceph-volume. 2026-02-20T22:24:24.641 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-677-gf7870454-1jammy_all.deb ... 2026-02-20T22:24:24.641 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:24.664 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-20T22:24:24.673 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:24.674 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:24.688 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-20T22:24:24.696 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:24.696 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:24.710 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-20T22:24:24.719 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:24.719 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:24.734 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package nvme-cli. 2026-02-20T22:24:24.743 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-20T22:24:24.743 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T22:24:24.775 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-20T22:24:24.784 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-20T22:24:24.784 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:24.800 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:24.805 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-20T22:24:24.813 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-20T22:24:24.822 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-20T22:24:24.823 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-20T22:24:24.837 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-pluggy. 2026-02-20T22:24:24.846 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-20T22:24:24.846 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-20T22:24:24.862 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-psutil. 2026-02-20T22:24:24.870 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-20T22:24:24.871 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-20T22:24:24.883 INFO:teuthology.orchestra.run.trial107.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T22:24:24.884 INFO:teuthology.orchestra.run.trial107.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-20T22:24:24.885 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-20T22:24:24.889 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-py. 2026-02-20T22:24:24.898 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-20T22:24:24.898 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-20T22:24:24.918 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-pygments. 2026-02-20T22:24:24.927 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-20T22:24:24.927 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T22:24:24.961 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-20T22:24:24.968 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-toml. 2026-02-20T22:24:24.977 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-20T22:24:24.978 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-20T22:24:24.992 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-pytest. 2026-02-20T22:24:25.001 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-20T22:24:25.002 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T22:24:25.023 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-simplejson. 2026-02-20T22:24:25.032 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-20T22:24:25.033 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-20T22:24:25.034 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:25.035 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-20T22:24:25.050 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package python3-webob. 2026-02-20T22:24:25.059 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-20T22:24:25.059 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T22:24:25.082 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-20T22:24:25.090 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-20T22:24:25.091 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-20T22:24:25.116 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-20T22:24:25.159 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package radosgw. 2026-02-20T22:24:25.167 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../84-radosgw_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:25.168 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:25.192 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-20T22:24:25.271 INFO:teuthology.orchestra.run.trial107.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T22:24:25.273 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-20T22:24:25.353 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T22:24:25.354 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T22:24:25.376 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package rbd-fuse. 2026-02-20T22:24:25.385 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:25.385 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:25.400 INFO:teuthology.orchestra.run.trial032.stdout:Selecting previously unselected package rbd-mirror. 2026-02-20T22:24:25.409 INFO:teuthology.orchestra.run.trial032.stdout:Preparing to unpack .../86-rbd-mirror_20.2.0-677-gf7870454-1jammy_amd64.deb ... 2026-02-20T22:24:25.410 INFO:teuthology.orchestra.run.trial032.stdout:Unpacking rbd-mirror (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:25.433 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T22:24:25.463 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-20T22:24:25.519 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-20T22:24:25.530 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-20T22:24:25.531 INFO:teuthology.orchestra.run.trial032.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-20T22:24:25.592 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:25.593 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T22:24:25.600 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-20T22:24:25.729 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-20T22:24:25.802 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:25.803 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-20T22:24:25.869 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-20T22:24:25.887 INFO:teuthology.orchestra.run.trial107.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:25.889 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-20T22:24:26.018 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T22:24:26.019 INFO:teuthology.orchestra.run.trial107.stdout:Setting up librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.020 INFO:teuthology.orchestra.run.trial107.stdout:Setting up librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.021 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.022 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T22:24:26.265 INFO:teuthology.orchestra.run.trial032.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-20T22:24:26.275 INFO:teuthology.orchestra.run.trial032.stdout:Setting up cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.317 INFO:teuthology.orchestra.run.trial032.stdout:Adding system user cephadm....done 2026-02-20T22:24:26.328 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-20T22:24:26.400 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:26.402 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-20T22:24:26.473 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-20T22:24:26.549 INFO:teuthology.orchestra.run.trial032.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-20T22:24:26.550 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-20T22:24:26.556 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.557 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.559 INFO:teuthology.orchestra.run.trial107.stdout:Setting up librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.560 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.560 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.637 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T22:24:26.638 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T22:24:26.645 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-20T22:24:26.769 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-20T22:24:26.844 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-20T22:24:26.918 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-ceph-argparse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:26.984 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-20T22:24:26.985 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-20T22:24:26.986 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-20T22:24:26.987 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-20T22:24:26.988 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-20T22:24:27.031 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.031 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.032 INFO:teuthology.orchestra.run.trial107.stdout:Setting up libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.034 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.035 INFO:teuthology.orchestra.run.trial107.stdout:Setting up rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.035 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.036 INFO:teuthology.orchestra.run.trial107.stdout:Setting up python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.037 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.066 INFO:teuthology.orchestra.run.trial107.stdout:Adding group ceph....done 2026-02-20T22:24:27.102 INFO:teuthology.orchestra.run.trial107.stdout:Adding system user ceph....done 2026-02-20T22:24:27.108 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-20T22:24:27.112 INFO:teuthology.orchestra.run.trial107.stdout:Setting system user ceph properties....done 2026-02-20T22:24:27.118 INFO:teuthology.orchestra.run.trial107.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-20T22:24:27.187 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libcephfs-proxy2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.188 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-20T22:24:27.205 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-20T22:24:27.266 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-20T22:24:27.353 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-20T22:24:27.488 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-20T22:24:27.607 INFO:teuthology.orchestra.run.trial032.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-20T22:24:27.608 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-20T22:24:27.704 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-20T22:24:27.835 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-20T22:24:27.880 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.881 INFO:teuthology.orchestra.run.trial107.stdout:Setting up radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:27.914 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-20T22:24:27.986 INFO:teuthology.orchestra.run.trial032.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-20T22:24:27.987 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:28.078 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-20T22:24:28.130 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T22:24:28.130 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T22:24:28.490 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:28.587 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:28.592 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-20T22:24:28.613 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-20T22:24:28.669 INFO:teuthology.orchestra.run.trial032.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-20T22:24:28.670 INFO:teuthology.orchestra.run.trial032.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-20T22:24:28.671 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-20T22:24:28.745 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-20T22:24:28.818 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:28.819 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-20T22:24:28.895 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-20T22:24:28.968 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-20T22:24:29.013 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.047 INFO:teuthology.orchestra.run.trial032.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-20T22:24:29.048 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-20T22:24:29.072 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T22:24:29.072 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T22:24:29.132 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-20T22:24:29.133 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-20T22:24:29.210 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-20T22:24:29.295 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-20T22:24:29.367 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:29.368 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-20T22:24:29.460 INFO:teuthology.orchestra.run.trial107.stdout:Setting up rbd-mirror (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.501 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-20T22:24:29.535 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T22:24:29.535 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T22:24:29.574 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-20T22:24:29.575 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-20T22:24:29.656 INFO:teuthology.orchestra.run.trial032.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-20T22:24:29.657 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-20T22:24:29.793 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-20T22:24:29.794 INFO:teuthology.orchestra.run.trial032.stdout:Setting up librados2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.795 INFO:teuthology.orchestra.run.trial032.stdout:Setting up librgw2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.796 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libsqlite3-mod-ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.797 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-20T22:24:29.883 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:29.964 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T22:24:29.964 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T22:24:30.326 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libcephfs2 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.327 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libradosstriper1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.328 INFO:teuthology.orchestra.run.trial032.stdout:Setting up librbd1 (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.329 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr-modules-core (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.330 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.343 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.401 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T22:24:30.401 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-20T22:24:30.436 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T22:24:30.436 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T22:24:30.800 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libcephfs-dev (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.801 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-rados (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.802 INFO:teuthology.orchestra.run.trial032.stdout:Setting up libcephfs-daemon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.803 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-rbd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.804 INFO:teuthology.orchestra.run.trial032.stdout:Setting up rbd-fuse (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.805 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-rgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.806 INFO:teuthology.orchestra.run.trial032.stdout:Setting up python3-cephfs (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.807 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-common (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.834 INFO:teuthology.orchestra.run.trial032.stdout:Adding group ceph....done 2026-02-20T22:24:30.840 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.841 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.852 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:30.869 INFO:teuthology.orchestra.run.trial032.stdout:Adding system user ceph....done 2026-02-20T22:24:30.880 INFO:teuthology.orchestra.run.trial032.stdout:Setting system user ceph properties....done 2026-02-20T22:24:30.885 INFO:teuthology.orchestra.run.trial032.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-20T22:24:30.922 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T22:24:30.923 INFO:teuthology.orchestra.run.trial107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T22:24:30.975 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-20T22:24:31.253 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-20T22:24:31.317 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.329 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.330 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.340 INFO:teuthology.orchestra.run.trial107.stdout:Setting up ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.448 INFO:teuthology.orchestra.run.trial107.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-20T22:24:31.535 INFO:teuthology.orchestra.run.trial107.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-20T22:24:31.628 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-test (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.629 INFO:teuthology.orchestra.run.trial032.stdout:Setting up radosgw (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:31.901 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T22:24:31.901 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-20T22:24:32.179 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.179 INFO:teuthology.orchestra.run.trial107.stdout:Running kernel seems to be up-to-date. 2026-02-20T22:24:32.179 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.179 INFO:teuthology.orchestra.run.trial107.stdout:The processor microcode seems to be up-to-date. 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout:No services need to be restarted. 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout:No containers need to be restarted. 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout:No user sessions are running outdated binaries. 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:32.180 INFO:teuthology.orchestra.run.trial107.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-20T22:24:32.297 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-base (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:32.415 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-20T22:24:32.805 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mds (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:32.864 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T22:24:32.864 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-20T22:24:32.982 INFO:teuthology.orchestra.run.trial107.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-20T22:24:32.991 DEBUG:teuthology.parallel:result is None 2026-02-20T22:24:33.262 INFO:teuthology.orchestra.run.trial032.stdout:Setting up rbd-mirror (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:33.337 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T22:24:33.338 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /lib/systemd/system/ceph-rbd-mirror.target. 2026-02-20T22:24:33.726 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:33.807 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T22:24:33.808 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-20T22:24:34.182 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-osd (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:34.273 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T22:24:34.274 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-20T22:24:34.664 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr-k8sevents (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:34.664 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:34.674 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mon (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:34.745 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T22:24:34.745 INFO:teuthology.orchestra.run.trial032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-20T22:24:35.134 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr-cephadm (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:35.144 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:35.145 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-mgr-dashboard (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:35.155 INFO:teuthology.orchestra.run.trial032.stdout:Setting up ceph-volume (20.2.0-677-gf7870454-1jammy) ... 2026-02-20T22:24:35.261 INFO:teuthology.orchestra.run.trial032.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-20T22:24:35.348 INFO:teuthology.orchestra.run.trial032.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-20T22:24:35.780 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.781 INFO:teuthology.orchestra.run.trial032.stdout:Running kernel seems to be up-to-date. 2026-02-20T22:24:35.781 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.781 INFO:teuthology.orchestra.run.trial032.stdout:The processor microcode seems to be up-to-date. 2026-02-20T22:24:35.781 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout:No services need to be restarted. 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout:No containers need to be restarted. 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout:No user sessions are running outdated binaries. 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:35.782 INFO:teuthology.orchestra.run.trial032.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-20T22:24:36.589 INFO:teuthology.orchestra.run.trial032.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-20T22:24:36.596 DEBUG:teuthology.parallel:result is None 2026-02-20T22:24:36.597 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:24:36.765 DEBUG:teuthology.orchestra.run.trial032:> dpkg-query -W -f '${Version}' ceph 2026-02-20T22:24:36.776 INFO:teuthology.orchestra.run.trial032.stdout:20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:36.777 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:36.777 INFO:teuthology.task.install:The correct ceph version 20.2.0-677-gf7870454-1jammy is installed. 2026-02-20T22:24:36.779 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:24:36.926 DEBUG:teuthology.orchestra.run.trial107:> dpkg-query -W -f '${Version}' ceph 2026-02-20T22:24:36.937 INFO:teuthology.orchestra.run.trial107.stdout:20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:36.938 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-677-gf7870454-1jammy 2026-02-20T22:24:36.938 INFO:teuthology.task.install:The correct ceph version 20.2.0-677-gf7870454-1jammy is installed. 2026-02-20T22:24:36.940 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-20T22:24:36.940 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:36.940 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T22:24:36.951 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:36.951 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-20T22:24:36.989 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-20T22:24:36.990 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:36.990 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T22:24:37.001 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T22:24:37.052 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:37.052 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/usr/bin/daemon-helper 2026-02-20T22:24:37.061 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-20T22:24:37.109 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-20T22:24:37.110 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:37.110 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T22:24:37.120 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T22:24:37.168 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:37.168 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-20T22:24:37.177 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-20T22:24:37.229 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-20T22:24:37.229 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:37.229 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T22:24:37.238 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T22:24:37.288 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:37.288 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/usr/bin/stdin-killer 2026-02-20T22:24:37.298 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-20T22:24:37.349 INFO:teuthology.run_tasks:Running task ceph... 2026-02-20T22:24:37.470 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2026-02-20T22:24:37.470 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod 777 /var/log/ceph 2026-02-20T22:24:37.471 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod 777 /var/log/ceph 2026-02-20T22:24:37.479 INFO:tasks.ceph:Disabling ceph logrotate... 2026-02-20T22:24:37.480 DEBUG:teuthology.orchestra.run.trial032:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T22:24:37.521 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -f -- /etc/logrotate.d/ceph 2026-02-20T22:24:37.530 INFO:tasks.ceph:Creating extra log directories... 2026-02-20T22:24:37.530 DEBUG:teuthology.orchestra.run.trial032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T22:24:37.570 DEBUG:teuthology.orchestra.run.trial107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2026-02-20T22:24:37.601 INFO:tasks.ceph:Creating ceph cluster cluster1... 2026-02-20T22:24:37.601 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-20T22:24:37.601 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587', 'branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '61587', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20260219.182737-tentacle'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7161, 'sha1': 'f7870454149f22574086dd2ff55d050585136531', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f7870454149f22574086dd2ff55d050585136531', 'targets': {'trial032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZKsPo+lvwLxn7Ee212rieC6zWJB+5QbSZDG9k7krFYPsVwEopXdvw5hQt2KBegIIEdhoyaRHkkH4ZdwY0vIO0=', 'trial107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOh8ioD937Osrcchl2HKEKdPFRr9QLyyACeqMcp5QXhMDbgV03LgBME8Q//ctKqZ6OIYcKsXBxkwhD+hdFR1J9c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-20_16:15:01', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-20T22:24:37.602 DEBUG:teuthology.orchestra.run.trial032:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-20T22:24:37.622 DEBUG:teuthology.orchestra.run.trial107:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2026-02-20T22:24:37.629 DEBUG:teuthology.orchestra.run.trial032:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T22:24:37.666 DEBUG:teuthology.orchestra.run.trial107:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T22:24:37.677 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:37.677 DEBUG:teuthology.orchestra.run.trial032:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T22:24:37.716 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T22:24:37.716 DEBUG:teuthology.orchestra.run.trial032:> stat /dev/vg_nvme/lv_1 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout:Device: 5h/5d Inode: 905 Links: 1 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout:Access: 2026-02-20 22:23:47.419807793 +0000 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout:Modify: 2026-02-20 22:23:47.255810582 +0000 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout:Change: 2026-02-20 22:23:47.255810582 +0000 2026-02-20T22:24:37.760 INFO:teuthology.orchestra.run.trial032.stdout: Birth: - 2026-02-20T22:24:37.760 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T22:24:37.807 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records in 2026-02-20T22:24:37.807 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records out 2026-02-20T22:24:37.807 INFO:teuthology.orchestra.run.trial032.stderr:512 bytes copied, 0.000192558 s, 2.7 MB/s 2026-02-20T22:24:37.808 DEBUG:teuthology.orchestra.run.trial032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T22:24:37.853 DEBUG:teuthology.orchestra.run.trial032:> stat /dev/vg_nvme/lv_2 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout:Device: 5h/5d Inode: 927 Links: 1 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout:Access: 2026-02-20 22:23:47.731802489 +0000 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout:Modify: 2026-02-20 22:23:47.575805142 +0000 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout:Change: 2026-02-20 22:23:47.575805142 +0000 2026-02-20T22:24:37.900 INFO:teuthology.orchestra.run.trial032.stdout: Birth: - 2026-02-20T22:24:37.900 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T22:24:37.947 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records in 2026-02-20T22:24:37.947 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records out 2026-02-20T22:24:37.947 INFO:teuthology.orchestra.run.trial032.stderr:512 bytes copied, 0.000180746 s, 2.8 MB/s 2026-02-20T22:24:37.948 DEBUG:teuthology.orchestra.run.trial032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T22:24:37.993 DEBUG:teuthology.orchestra.run.trial032:> stat /dev/vg_nvme/lv_3 2026-02-20T22:24:38.040 INFO:teuthology.orchestra.run.trial032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T22:24:38.040 INFO:teuthology.orchestra.run.trial032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:24:38.040 INFO:teuthology.orchestra.run.trial032.stdout:Device: 5h/5d Inode: 953 Links: 1 2026-02-20T22:24:38.041 INFO:teuthology.orchestra.run.trial032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:24:38.041 INFO:teuthology.orchestra.run.trial032.stdout:Access: 2026-02-20 22:23:48.091796369 +0000 2026-02-20T22:24:38.041 INFO:teuthology.orchestra.run.trial032.stdout:Modify: 2026-02-20 22:23:47.903799566 +0000 2026-02-20T22:24:38.041 INFO:teuthology.orchestra.run.trial032.stdout:Change: 2026-02-20 22:23:47.903799566 +0000 2026-02-20T22:24:38.041 INFO:teuthology.orchestra.run.trial032.stdout: Birth: - 2026-02-20T22:24:38.041 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T22:24:38.091 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records in 2026-02-20T22:24:38.091 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records out 2026-02-20T22:24:38.091 INFO:teuthology.orchestra.run.trial032.stderr:512 bytes copied, 0.000173502 s, 3.0 MB/s 2026-02-20T22:24:38.092 DEBUG:teuthology.orchestra.run.trial032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T22:24:38.137 DEBUG:teuthology.orchestra.run.trial032:> stat /dev/vg_nvme/lv_4 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout:Device: 5h/5d Inode: 974 Links: 1 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout:Access: 2026-02-20 22:23:48.247793716 +0000 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout:Modify: 2026-02-20 22:23:48.243793785 +0000 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout:Change: 2026-02-20 22:23:48.243793785 +0000 2026-02-20T22:24:38.184 INFO:teuthology.orchestra.run.trial032.stdout: Birth: - 2026-02-20T22:24:38.184 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T22:24:38.231 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records in 2026-02-20T22:24:38.231 INFO:teuthology.orchestra.run.trial032.stderr:1+0 records out 2026-02-20T22:24:38.231 INFO:teuthology.orchestra.run.trial032.stderr:512 bytes copied, 0.000167371 s, 3.1 MB/s 2026-02-20T22:24:38.232 DEBUG:teuthology.orchestra.run.trial032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T22:24:38.278 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:24:38.278 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial032.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T22:24:38.278 INFO:tasks.ceph:Generating config... 2026-02-20T22:24:38.280 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client] log to stderr = False 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.281 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.282 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.282 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.282 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.282 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-20T22:24:38.282 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-20T22:24:38.282 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-20T22:24:38.282 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-20T22:24:38.282 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-20T22:24:38.282 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-20T22:24:38.282 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-20T22:24:38.282 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-20T22:24:38.282 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-20T22:24:38.283 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-20T22:24:38.284 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-20T22:24:38.284 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-20T22:24:38.284 INFO:tasks.ceph:Setting up cluster1.mon.a... 2026-02-20T22:24:38.284 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.337 INFO:teuthology.orchestra.run.trial032.stdout:creating /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.340 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.400 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.448 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '10.20.193.32')] 2026-02-20T22:24:38.449 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.32', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-20T22:24:38.450 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:38.451 DEBUG:teuthology.orchestra.run.trial032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T22:24:38.493 DEBUG:teuthology.orchestra.run.trial032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.32 --print /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:38.553 INFO:teuthology.orchestra.run.trial032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:38.553 INFO:teuthology.orchestra.run.trial032.stdout:monmaptool: generated fsid d1a0e502-1b23-4a49-a64f-9ce85d3de0ff 2026-02-20T22:24:38.553 INFO:teuthology.orchestra.run.trial032.stdout:setting min_mon_release = tentacle 2026-02-20T22:24:38.553 INFO:teuthology.orchestra.run.trial032.stdout:epoch 0 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:fsid d1a0e502-1b23-4a49-a64f-9ce85d3de0ff 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:last_changed 2026-02-20T22:24:38.551440+0000 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:created 2026-02-20T22:24:38.551440+0000 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:min_mon_release 20 (tentacle) 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:election_strategy: 1 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:0: [v2:10.20.193.32:3300/0,v1:10.20.193.32:6789/0] mon.a 2026-02-20T22:24:38.554 INFO:teuthology.orchestra.run.trial032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2026-02-20T22:24:38.556 DEBUG:teuthology.orchestra.run.trial032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T22:24:38.600 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID d1a0e502-1b23-4a49-a64f-9ce85d3de0ff... 2026-02-20T22:24:38.602 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-20T22:24:38.641 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2026-02-20T22:24:38.660 INFO:teuthology.orchestra.run.trial032.stdout:[global] 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: chdir = "" 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: auth supported = cephx 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: filestore xattr use omap = true 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: mon clock drift allowed = 1.000 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: osd crush chooseleaf type = 0 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: auth debug = true 2026-02-20T22:24:38.661 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: ms die on old message = true 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: ms die on bug = true 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: mon pg warn max object skew = 0 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: osd pool default size = 2 2026-02-20T22:24:38.662 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon osd allow primary affinity = true 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon osd allow pg remap = true 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on legacy crush tunables = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on crush straw calc version zero = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on no sortbitwise = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on osd down out interval zero = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on too few osds = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: mon_allow_pool_size_one = true 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.663 INFO:teuthology.orchestra.run.trial032.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: osd default data pool replay window = 5 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon allow pool delete = true 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon cluster log file level = debug 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: debug asserts on shutdown = true 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon health detail to clog = false 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon host = 10.20.193.32 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon client directed command retry = 5 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on pool no app = False 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: ms inject socket failures = 5000 2026-02-20T22:24:38.664 INFO:teuthology.orchestra.run.trial032.stdout: fsid = d1a0e502-1b23-4a49-a64f-9ce85d3de0ff 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout:[osd] 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd journal size = 100 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd scrub load threshold = 5.0 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd scrub max interval = 600 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd mclock profile = high_recovery_ops 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd mclock skip benchmark = true 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd recover clone overlap = true 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: osd recovery max chunk = 1048576 2026-02-20T22:24:38.665 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd debug shutdown = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd debug op order = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd debug verify stray on activate = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd debug trim objects = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd open classes on start = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd debug pg log writeout = true 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd deep scrub update digest min age = 30 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: osd map max advance = 10 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.666 INFO:teuthology.orchestra.run.trial032.stdout: journal zero on create = true 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: filestore ondisk finisher threads = 3 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: filestore apply finisher threads = 3 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bdev debug aio = true 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: osd debug misdirected ops = true 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bdev async discard = True 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bdev enable discard = True 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bluestore allocator = stupid 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bluestore block size = 96636764160 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: bluestore fsck on mount = True 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: debug bluefs = 1/20 2026-02-20T22:24:38.667 INFO:teuthology.orchestra.run.trial032.stdout: debug bluestore = 1/20 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug osd = 20 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug rocksdb = 4/10 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: mon osd backfillfull_ratio = 0.85 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: mon osd full ratio = 0.9 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: mon osd nearfull ratio = 0.8 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: osd failsafe full ratio = 0.95 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: osd objectstore = bluestore 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout:[mgr] 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug mgr = 20 2026-02-20T22:24:38.668 INFO:teuthology.orchestra.run.trial032.stdout: debug mon = 20 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: debug auth = 20 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: mgr/telemetry/nag = false 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout:[mon] 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: debug mon = 20 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: debug paxos = 20 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: debug auth = 20 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: mon data avail warn = 5 2026-02-20T22:24:38.669 INFO:teuthology.orchestra.run.trial032.stdout: mon mgr mkfs grace = 240 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon osd reporter subtree level = osd 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon osd prime pg temp = true 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T22:24:38.670 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: # 1m isn't quite enough 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: mon_down_mkfs_grace = 2m 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_filestore_osds = false 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout:[client] 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: rgw cache enabled = true 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: rgw enable ops log = true 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: rgw enable usage log = true 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T22:24:38.671 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: debug rbd = 20 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: debug rbd_mirror = 15 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: log to stderr = False 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: rbd default features = 125 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: rbd mirror image policy type = simple 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout:[mon.a] 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.0] 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.1] 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.672 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.2] 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.3] 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.673 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout:[global] 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: chdir = "" 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: auth supported = cephx 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: filestore xattr use omap = true 2026-02-20T22:24:38.675 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: mon clock drift allowed = 1.000 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: osd crush chooseleaf type = 0 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: auth debug = true 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: ms die on old message = true 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: ms die on bug = true 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: mon pg warn max object skew = 0 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.676 INFO:teuthology.orchestra.run.trial107.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: osd pool default size = 2 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon osd allow primary affinity = true 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon osd allow pg remap = true 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on legacy crush tunables = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on crush straw calc version zero = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on no sortbitwise = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on osd down out interval zero = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on too few osds = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T22:24:38.677 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon_allow_pool_size_one = true 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: osd default data pool replay window = 5 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon allow pool delete = true 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon cluster log file level = debug 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: debug asserts on shutdown = true 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon health detail to clog = false 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon host = 10.20.193.32 2026-02-20T22:24:38.678 INFO:teuthology.orchestra.run.trial107.stdout: mon client directed command retry = 5 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on pool no app = False 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: ms inject socket failures = 5000 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: fsid = d1a0e502-1b23-4a49-a64f-9ce85d3de0ff 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout:[osd] 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: osd journal size = 100 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: osd scrub load threshold = 5.0 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: osd scrub max interval = 600 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: osd mclock profile = high_recovery_ops 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: osd mclock skip benchmark = true 2026-02-20T22:24:38.679 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd recover clone overlap = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd recovery max chunk = 1048576 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd debug shutdown = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd debug op order = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd debug verify stray on activate = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd debug trim objects = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd open classes on start = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd debug pg log writeout = true 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd deep scrub update digest min age = 30 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.680 INFO:teuthology.orchestra.run.trial107.stdout: osd map max advance = 10 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: journal zero on create = true 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: filestore ondisk finisher threads = 3 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: filestore apply finisher threads = 3 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: bdev debug aio = true 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: osd debug misdirected ops = true 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: bdev async discard = True 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: bdev enable discard = True 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: bluestore allocator = stupid 2026-02-20T22:24:38.681 INFO:teuthology.orchestra.run.trial107.stdout: bluestore block size = 96636764160 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: bluestore fsck on mount = True 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: debug bluefs = 1/20 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: debug bluestore = 1/20 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: debug osd = 20 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: debug rocksdb = 4/10 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: mon osd backfillfull_ratio = 0.85 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: mon osd full ratio = 0.9 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: mon osd nearfull ratio = 0.8 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: osd failsafe full ratio = 0.95 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: osd objectstore = bluestore 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.682 INFO:teuthology.orchestra.run.trial107.stdout:[mgr] 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug mgr = 20 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug mon = 20 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug auth = 20 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: mgr/telemetry/nag = false 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout:[mon] 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug mon = 20 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug paxos = 20 2026-02-20T22:24:38.683 INFO:teuthology.orchestra.run.trial107.stdout: debug auth = 20 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon data avail warn = 5 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon mgr mkfs grace = 240 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon osd reporter subtree level = osd 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon osd prime pg temp = true 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T22:24:38.684 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: # 1m isn't quite enough 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: mon_down_mkfs_grace = 2m 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_filestore_osds = false 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout:[client] 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: rgw cache enabled = true 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: rgw enable ops log = true 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: rgw enable usage log = true 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T22:24:38.685 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: debug rbd = 20 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: debug rbd_mirror = 15 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: log to stderr = False 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: rbd default features = 125 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: rbd mirror image policy type = simple 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout:[mon.a] 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.0] 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.1] 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.686 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.2] 2026-02-20T22:24:38.687 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.687 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.687 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.3] 2026-02-20T22:24:38.687 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:24:38.687 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:24:38.687 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2026-02-20T22:24:38.687 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.738 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-20T22:24:38.738 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:38.738 DEBUG:teuthology.orchestra.run.trial032:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2026-02-20T22:24:38.785 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:38.785 DEBUG:teuthology.orchestra.run.trial032:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2026-02-20T22:24:38.831 INFO:tasks.ceph:Sending monmap to node ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:24:38.831 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:38.831 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-20T22:24:38.831 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.886 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:38.886 DEBUG:teuthology.orchestra.run.trial032:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:38.932 INFO:tasks.ceph:Sending monmap to node ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:24:38.932 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:38.933 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/etc/ceph/cluster1.keyring 2026-02-20T22:24:38.933 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2026-02-20T22:24:38.947 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:38.947 DEBUG:teuthology.orchestra.run.trial107:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:38.994 INFO:tasks.ceph:Setting up mon nodes... 2026-02-20T22:24:38.994 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-20T22:24:38.994 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-20T22:24:39.024 INFO:teuthology.orchestra.run.trial032.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2026-02-20T22:24:39.027 INFO:tasks.ceph:Setting up mds nodes... 2026-02-20T22:24:39.028 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-20T22:24:39.028 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2026-02-20T22:24:39.091 INFO:teuthology.orchestra.run.trial032.stdout:creating /etc/ceph/cluster1.client.0.keyring 2026-02-20T22:24:39.099 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2026-02-20T22:24:39.125 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2026-02-20T22:24:39.134 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-20T22:24:39.200 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2026-02-20T22:24:39.208 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-20T22:24:39.272 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2026-02-20T22:24:39.281 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-20T22:24:39.344 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2026-02-20T22:24:39.351 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-20T22:24:39.415 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2026-02-20T22:24:39.423 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-20T22:24:39.487 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2026-02-20T22:24:39.497 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-20T22:24:39.559 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2026-02-20T22:24:39.567 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-20T22:24:39.635 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2026-02-20T22:24:39.643 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-20T22:24:39.643 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial032.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T22:24:39.644 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2026-02-20T22:24:39.653 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:24:39.653 INFO:tasks.ceph:role: cluster1.osd.0 2026-02-20T22:24:39.653 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:24:39.654 DEBUG:teuthology.orchestra.run.trial032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T22:24:39.706 INFO:teuthology.orchestra.run.trial032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:24:39.706 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:24:39.706 INFO:teuthology.orchestra.run.trial032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:24:39.707 INFO:teuthology.orchestra.run.trial032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:24:40.284 INFO:teuthology.orchestra.run.trial032.stdout:Discarding blocks...Done. 2026-02-20T22:24:40.286 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial032.front.sepia.ceph.com -o noatime 2026-02-20T22:24:40.286 DEBUG:teuthology.orchestra.run.trial032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2026-02-20T22:24:40.341 DEBUG:teuthology.orchestra.run.trial032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2026-02-20T22:24:40.390 INFO:teuthology.orchestra.run.trial032.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:24:40.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:24:40.391 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2026-02-20T22:24:40.440 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:24:40.441 INFO:tasks.ceph:role: cluster1.osd.1 2026-02-20T22:24:40.441 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:24:40.441 DEBUG:teuthology.orchestra.run.trial032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T22:24:40.494 INFO:teuthology.orchestra.run.trial032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:24:40.494 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:24:40.494 INFO:teuthology.orchestra.run.trial032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:24:40.495 INFO:teuthology.orchestra.run.trial032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:24:41.060 INFO:teuthology.orchestra.run.trial032.stdout:Discarding blocks...Done. 2026-02-20T22:24:41.062 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial032.front.sepia.ceph.com -o noatime 2026-02-20T22:24:41.062 DEBUG:teuthology.orchestra.run.trial032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2026-02-20T22:24:41.121 DEBUG:teuthology.orchestra.run.trial032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2026-02-20T22:24:41.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:24:41.171 INFO:teuthology.orchestra.run.trial032.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:24:41.171 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2026-02-20T22:24:41.220 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:24:41.220 INFO:tasks.ceph:role: cluster1.osd.2 2026-02-20T22:24:41.220 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:24:41.221 DEBUG:teuthology.orchestra.run.trial032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:24:41.274 INFO:teuthology.orchestra.run.trial032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:24:41.843 INFO:teuthology.orchestra.run.trial032.stdout:Discarding blocks...Done. 2026-02-20T22:24:41.844 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial032.front.sepia.ceph.com -o noatime 2026-02-20T22:24:41.844 DEBUG:teuthology.orchestra.run.trial032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2026-02-20T22:24:41.896 DEBUG:teuthology.orchestra.run.trial032:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2026-02-20T22:24:41.941 INFO:teuthology.orchestra.run.trial032.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:24:41.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:24:41.942 DEBUG:teuthology.orchestra.run.trial032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:42.005 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.003+0000 7f774479da40 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2026-02-20T22:24:42.006 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.003+0000 7f774479da40 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2026-02-20T22:24:42.006 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.003+0000 7f774479da40 -1 bdev(0x5557ae349800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2026-02-20T22:24:42.006 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.003+0000 7f774479da40 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2026-02-20T22:24:42.866 DEBUG:teuthology.orchestra.run.trial032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2026-02-20T22:24:42.917 DEBUG:teuthology.orchestra.run.trial032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:42.981 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.979+0000 7efedfc0aa40 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2026-02-20T22:24:42.982 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.979+0000 7efedfc0aa40 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2026-02-20T22:24:42.982 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.979+0000 7efedfc0aa40 -1 bdev(0x55ae0090f800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2026-02-20T22:24:42.982 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:42.979+0000 7efedfc0aa40 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2026-02-20T22:24:43.880 DEBUG:teuthology.orchestra.run.trial032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2026-02-20T22:24:43.929 DEBUG:teuthology.orchestra.run.trial032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:43.994 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:43.991+0000 7f2668f02a40 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2026-02-20T22:24:43.994 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:43.991+0000 7f2668f02a40 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2026-02-20T22:24:43.994 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:43.991+0000 7f2668f02a40 -1 bdev(0x561afb13d800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2026-02-20T22:24:43.994 INFO:teuthology.orchestra.run.trial032.stderr:2026-02-20T22:24:43.991+0000 7f2668f02a40 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2026-02-20T22:24:44.849 DEBUG:teuthology.orchestra.run.trial032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2026-02-20T22:24:44.901 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-20T22:24:44.902 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:44.902 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2026-02-20T22:24:44.953 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:44.953 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2026-02-20T22:24:45.005 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:45.005 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2026-02-20T22:24:45.057 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:45.057 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2026-02-20T22:24:45.109 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:45.109 DEBUG:teuthology.orchestra.run.trial032:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2026-02-20T22:24:45.156 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.156 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2026-02-20T22:24:45.161 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.161 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2026-02-20T22:24:45.206 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.206 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2026-02-20T22:24:45.254 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.254 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2026-02-20T22:24:45.300 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.301 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2026-02-20T22:24:45.349 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.349 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2026-02-20T22:24:45.393 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.393 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2026-02-20T22:24:45.437 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:24:45.437 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2026-02-20T22:24:45.480 INFO:tasks.ceph:Adding keys to all mons... 2026-02-20T22:24:45.481 DEBUG:teuthology.orchestra.run.trial032:> sudo tee -a /etc/ceph/cluster1.keyring 2026-02-20T22:24:45.489 INFO:teuthology.orchestra.run.trial032.stdout:[mgr.x] 2026-02-20T22:24:45.489 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpdzZiARAAaeFloq9olfytzRG1wTRfVg== 2026-02-20T22:24:45.489 INFO:teuthology.orchestra.run.trial032.stdout:[osd.0] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAq35hppjhCABAA0HPvbdGsLvMbvcVHPqmr5A== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[osd.1] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAq35hpsRlnOhAAUCmWMtlX4isxnYsFvWuQXw== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[osd.2] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAr35hpnUYcOxAAKhOYlLxsn6RgZDNj6+62GA== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[client.0] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpkf5cBRAAt4lBSaCiEa9C9Ozhqa2XaQ== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpaCRiBxAAJZFxT5FUm/nncx38BKBFVw== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.0] 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hp8d3iCxAAgEh2Zh1h0n1PjWXqVLxVrA== 2026-02-20T22:24:45.490 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.1] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpuZUkEBAAz/mv65oyNEOsWR4ynt0wnA== 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.2] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpIF9sFBAA4wQvsv+eUTA7KtXjHpaVkQ== 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.3] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpj++yGBAA4vHEQChCCthgYPQ8cMM1eA== 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.4] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpWrn6HBAAA5BEupuTd5Bkqd8blZl5ow== 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.5] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hpwQxHIRAA7INWIRZzvV5V20iZmpmYfw== 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.6] 2026-02-20T22:24:45.491 INFO:teuthology.orchestra.run.trial032.stdout: key = AQAn35hptX3NJRAAT6qKVhSOlSMGu5Gzm3wCVQ== 2026-02-20T22:24:45.493 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-20T22:24:45.558 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:24:45.621 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:24:45.685 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:24:45.749 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:45.814 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:45.878 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:45.942 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.007 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.070 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.134 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.197 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.262 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:24:46.326 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-20T22:24:46.326 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2026-02-20T22:24:46.376 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2026-02-20T22:24:46.446 DEBUG:teuthology.orchestra.run.trial032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2026-02-20T22:24:46.496 DEBUG:teuthology.orchestra.run.trial032:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2026-02-20T22:24:46.539 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2026-02-20T22:24:46.540 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2026-02-20T22:24:46.540 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2026-02-20T22:24:46.582 INFO:tasks.ceph.cluster1.mon.a:Started 2026-02-20T22:24:46.582 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2026-02-20T22:24:46.583 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2026-02-20T22:24:46.583 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2026-02-20T22:24:46.585 INFO:tasks.ceph.cluster1.mgr.x:Started 2026-02-20T22:24:46.585 DEBUG:tasks.ceph:set 0 configs 2026-02-20T22:24:46.585 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 config dump 2026-02-20T22:24:46.695 INFO:teuthology.orchestra.run.trial032.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-20T22:24:46.705 INFO:tasks.ceph:Setting crush tunables to default 2026-02-20T22:24:46.706 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 osd crush tunables default 2026-02-20T22:24:46.805 INFO:teuthology.orchestra.run.trial032.stderr:adjusted tunables profile to default 2026-02-20T22:24:46.814 INFO:tasks.ceph:check_enable_crimson: False 2026-02-20T22:24:46.814 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2026-02-20T22:24:46.815 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:46.815 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2026-02-20T22:24:46.825 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:46.825 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2026-02-20T22:24:46.880 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:24:46.880 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2026-02-20T22:24:46.928 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 osd new 9ca3162b-8d73-4cac-a717-cf31c5b51d87 0 2026-02-20T22:24:47.066 INFO:teuthology.orchestra.run.trial032.stdout:0 2026-02-20T22:24:47.076 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 osd new 65fbbb8d-e0b9-4e23-8050-11c3774a6bd9 1 2026-02-20T22:24:47.174 INFO:teuthology.orchestra.run.trial032.stdout:1 2026-02-20T22:24:47.183 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 osd new 02f1be14-e4b6-4854-8214-b3c8094dc992 2 2026-02-20T22:24:47.282 INFO:teuthology.orchestra.run.trial032.stdout:2 2026-02-20T22:24:47.291 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2026-02-20T22:24:47.291 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2026-02-20T22:24:47.293 INFO:tasks.ceph.cluster1.osd.0:Started 2026-02-20T22:24:47.293 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2026-02-20T22:24:47.294 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2026-02-20T22:24:47.295 INFO:tasks.ceph.cluster1.osd.1:Started 2026-02-20T22:24:47.296 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2026-02-20T22:24:47.296 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2026-02-20T22:24:47.298 INFO:tasks.ceph.cluster1.osd.2:Started 2026-02-20T22:24:47.298 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-20T22:24:47.395 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:47.396 INFO:teuthology.orchestra.run.trial032.stdout:{"epoch":5,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T22:24:47.280068+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:24:47.405 INFO:tasks.ceph.ceph_manager.cluster1:[] 2026-02-20T22:24:47.405 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-20T22:24:47.707 DEBUG:teuthology.orchestra.run.trial032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-20T22:24:47.796 INFO:teuthology.misc.health.trial032.stdout: 2026-02-20T22:24:47.796 INFO:teuthology.misc.health.trial032.stdout:{"epoch":5,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T22:24:47.280068+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:24:47.804 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-20T22:24:47.810 INFO:tasks.ceph.cluster1.osd.0.trial032.stderr:2026-02-20T22:24:47.807+0000 7f5bb2d7aa40 -1 Falling back to public interface 2026-02-20T22:24:47.826 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:24:47.823+0000 7fa6fe991a40 -1 Falling back to public interface 2026-02-20T22:24:47.826 INFO:tasks.ceph.cluster1.osd.2.trial032.stderr:2026-02-20T22:24:47.823+0000 7f9b05164a40 -1 Falling back to public interface 2026-02-20T22:24:48.187 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:ERROR:root:Module 'jmespath' is not installed. 2026-02-20T22:24:48.240 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:ERROR:dashboard.controllers.nvmeof:Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) 2026-02-20T22:24:48.245 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:ERROR:sso:Module 'jmespath' is not installed. 2026-02-20T22:24:48.293 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-20T22:24:48.735 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-20T22:24:48.735 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-20T22:24:48.736 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr: from numpy import show_config as show_numpy_config 2026-02-20T22:24:49.105 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:24:49.099+0000 7fa6fe991a40 -1 osd.1 0 log_to_monitors true 2026-02-20T22:24:49.136 INFO:tasks.ceph.cluster1.osd.0.trial032.stderr:2026-02-20T22:24:49.131+0000 7f5bb2d7aa40 -1 osd.0 0 log_to_monitors true 2026-02-20T22:24:49.137 INFO:tasks.ceph.cluster1.osd.2.trial032.stderr:2026-02-20T22:24:49.135+0000 7f9b05164a40 -1 osd.2 0 log_to_monitors true 2026-02-20T22:24:49.814 INFO:tasks.ceph.cluster1.mgr.x.trial032.stderr:2026-02-20T22:24:49.811+0000 7f4d76181640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-20T22:24:50.885 INFO:tasks.ceph.cluster1.osd.0.trial032.stderr:2026-02-20T22:24:50.883+0000 7f5bae500640 -1 osd.0 0 waiting for initial osdmap 2026-02-20T22:24:50.885 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:24:50.883+0000 7fa6fa117640 -1 osd.1 0 waiting for initial osdmap 2026-02-20T22:24:50.886 INFO:tasks.ceph.cluster1.osd.2.trial032.stderr:2026-02-20T22:24:50.883+0000 7f9b008ea640 -1 osd.2 0 waiting for initial osdmap 2026-02-20T22:24:50.888 INFO:tasks.ceph.cluster1.osd.0.trial032.stderr:2026-02-20T22:24:50.883+0000 7f5ba6319640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:24:50.888 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:24:50.883+0000 7fa6f1f30640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:24:50.888 INFO:tasks.ceph.cluster1.osd.2.trial032.stderr:2026-02-20T22:24:50.883+0000 7f9af8703640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:24:54.107 DEBUG:teuthology.orchestra.run.trial032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2026-02-20T22:24:54.255 INFO:teuthology.misc.health.trial032.stdout: 2026-02-20T22:24:54.255 INFO:teuthology.misc.health.trial032.stdout:{"epoch":10,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T22:24:53.866269+0000","last_up_change":"2026-02-20T22:24:51.859134+0000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:24:52.849512+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6800","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6801","nonce":3011236143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6802","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6803","nonce":3011236143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6806","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6807","nonce":3011236143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6804","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6805","nonce":3011236143}]},"public_addr":"10.20.193.32:6801/3011236143","cluster_addr":"10.20.193.32:6803/3011236143","heartbeat_back_addr":"10.20.193.32:6807/3011236143","heartbeat_front_addr":"10.20.193.32:6805/3011236143","state":["exists","up"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6809","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6812","nonce":1911326603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6815","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6817","nonce":1911326603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6822","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6823","nonce":1911326603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6820","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6821","nonce":1911326603}]},"public_addr":"10.20.193.32:6812/1911326603","cluster_addr":"10.20.193.32:6817/1911326603","heartbeat_back_addr":"10.20.193.32:6823/1911326603","heartbeat_front_addr":"10.20.193.32:6821/1911326603","state":["exists","up"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6808","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6810","nonce":2685558554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6811","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6813","nonce":2685558554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6818","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6819","nonce":2685558554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6814","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6816","nonce":2685558554}]},"public_addr":"10.20.193.32:6810/2685558554","cluster_addr":"10.20.193.32:6813/2685558554","heartbeat_back_addr":"10.20.193.32:6819/2685558554","heartbeat_front_addr":"10.20.193.32:6816/2685558554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:24:54.264 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-20T22:24:54.264 INFO:tasks.ceph:Creating RBD pool 2026-02-20T22:24:54.265 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 osd pool create rbd 8 2026-02-20T22:24:54.874 INFO:teuthology.orchestra.run.trial032.stderr:pool 'rbd' created 2026-02-20T22:24:54.886 DEBUG:teuthology.orchestra.run.trial032:> rbd --cluster cluster1 pool init rbd 2026-02-20T22:24:57.889 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2026-02-20T22:24:57.890 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 config log 1 --format=json 2026-02-20T22:24:57.891 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-20T22:24:58.052 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:58.065 INFO:teuthology.orchestra.run.trial032.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-20T22:24:58.065 INFO:tasks.ceph_manager:config epoch is 1 2026-02-20T22:24:58.065 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2026-02-20T22:24:58.065 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2026-02-20T22:24:58.066 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2026-02-20T22:24:58.245 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:58.255 INFO:teuthology.orchestra.run.trial032.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6824","nonce":2848210690},{"type":"v1","addr":"10.20.193.32:6825","nonce":2848210690}]},"active_addr":"10.20.193.32:6825/2848210690","active_change":"2026-02-20T22:24:48.804109+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":77268015}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":2147531542}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":1581566026}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":2857531857}]}]} 2026-02-20T22:24:58.259 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2026-02-20T22:24:58.259 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2026-02-20T22:24:58.260 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-20T22:24:58.417 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:58.417 INFO:teuthology.orchestra.run.trial032.stdout:{"epoch":14,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T22:24:57.877073+0000","last_up_change":"2026-02-20T22:24:51.859134+0000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:24:52.849512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:24:54.413552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6800","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6801","nonce":3011236143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6802","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6803","nonce":3011236143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6806","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6807","nonce":3011236143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6804","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6805","nonce":3011236143}]},"public_addr":"10.20.193.32:6801/3011236143","cluster_addr":"10.20.193.32:6803/3011236143","heartbeat_back_addr":"10.20.193.32:6807/3011236143","heartbeat_front_addr":"10.20.193.32:6805/3011236143","state":["exists","up"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6809","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6812","nonce":1911326603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6815","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6817","nonce":1911326603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6822","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6823","nonce":1911326603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6820","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6821","nonce":1911326603}]},"public_addr":"10.20.193.32:6812/1911326603","cluster_addr":"10.20.193.32:6817/1911326603","heartbeat_back_addr":"10.20.193.32:6823/1911326603","heartbeat_front_addr":"10.20.193.32:6821/1911326603","state":["exists","up"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6808","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6810","nonce":2685558554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6811","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6813","nonce":2685558554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6818","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6819","nonce":2685558554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6814","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6816","nonce":2685558554}]},"public_addr":"10.20.193.32:6810/2685558554","cluster_addr":"10.20.193.32:6813/2685558554","heartbeat_back_addr":"10.20.193.32:6819/2685558554","heartbeat_front_addr":"10.20.193.32:6816/2685558554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:24:58.428 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2026-02-20T22:24:58.429 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-20T22:24:58.581 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:24:58.581 INFO:teuthology.orchestra.run.trial032.stdout:{"epoch":14,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T22:24:57.877073+0000","last_up_change":"2026-02-20T22:24:51.859134+0000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:24:52.849512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:24:54.413552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6800","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6801","nonce":3011236143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6802","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6803","nonce":3011236143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6806","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6807","nonce":3011236143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6804","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6805","nonce":3011236143}]},"public_addr":"10.20.193.32:6801/3011236143","cluster_addr":"10.20.193.32:6803/3011236143","heartbeat_back_addr":"10.20.193.32:6807/3011236143","heartbeat_front_addr":"10.20.193.32:6805/3011236143","state":["exists","up"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6809","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6812","nonce":1911326603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6815","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6817","nonce":1911326603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6822","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6823","nonce":1911326603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6820","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6821","nonce":1911326603}]},"public_addr":"10.20.193.32:6812/1911326603","cluster_addr":"10.20.193.32:6817/1911326603","heartbeat_back_addr":"10.20.193.32:6823/1911326603","heartbeat_front_addr":"10.20.193.32:6821/1911326603","state":["exists","up"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6808","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6810","nonce":2685558554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6811","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6813","nonce":2685558554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6818","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6819","nonce":2685558554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6814","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6816","nonce":2685558554}]},"public_addr":"10.20.193.32:6810/2685558554","cluster_addr":"10.20.193.32:6813/2685558554","heartbeat_back_addr":"10.20.193.32:6819/2685558554","heartbeat_front_addr":"10.20.193.32:6816/2685558554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:24:58.592 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2026-02-20T22:24:58.593 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2026-02-20T22:24:58.593 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2026-02-20T22:24:58.677 INFO:teuthology.orchestra.run.trial032.stdout:34359738372 2026-02-20T22:24:58.677 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-20T22:24:58.679 INFO:teuthology.orchestra.run.trial032.stdout:34359738371 2026-02-20T22:24:58.680 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-20T22:24:58.691 INFO:teuthology.orchestra.run.trial032.stdout:34359738371 2026-02-20T22:24:58.691 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2026-02-20T22:24:58.834 INFO:teuthology.orchestra.run.trial032.stdout:34359738369 2026-02-20T22:24:58.839 INFO:teuthology.orchestra.run.trial032.stdout:34359738370 2026-02-20T22:24:58.846 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738369 for osd.1 2026-02-20T22:24:58.852 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738370 for osd.2 2026-02-20T22:24:58.887 INFO:teuthology.orchestra.run.trial032.stdout:34359738371 2026-02-20T22:24:58.898 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.0 2026-02-20T22:24:58.898 DEBUG:teuthology.parallel:result is None 2026-02-20T22:24:59.847 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2026-02-20T22:24:59.853 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2026-02-20T22:25:00.008 INFO:teuthology.orchestra.run.trial032.stdout:34359738371 2026-02-20T22:25:00.019 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738371 got 34359738371 for osd.1 2026-02-20T22:25:00.019 DEBUG:teuthology.parallel:result is None 2026-02-20T22:25:00.048 INFO:teuthology.orchestra.run.trial032.stdout:34359738372 2026-02-20T22:25:00.060 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738372 got 34359738372 for osd.2 2026-02-20T22:25:00.060 DEBUG:teuthology.parallel:result is None 2026-02-20T22:25:00.060 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-20T22:25:00.060 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T22:25:00.255 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:00.256 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T22:25:00.265 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-20T22:24:58.809609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82400,"kb_used_data":1520,"kb_used_omap":23,"kb_used_meta":80808,"kb_avail":283033120,"statfs":{"total":289910292480,"available":289825914880,"internally_reserved":0,"allocated":1556480,"data_stored":1283458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23725,"internal_metadata":82748243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.926096"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880876+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T22:24:57.880876+0000","last_peered":"2026-02-20T22:24:57.880876+0000","last_clean":"2026-02-20T22:24:57.880876+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T22:24:57.880876+0000","last_undegraded":"2026-02-20T22:24:57.880876+0000","last_fullsized":"2026-02-20T22:24:57.880876+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880402+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T22:24:57.880402+0000","last_peered":"2026-02-20T22:24:57.880402+0000","last_clean":"2026-02-20T22:24:57.880402+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T22:24:57.880402+0000","last_undegraded":"2026-02-20T22:24:57.880402+0000","last_fullsized":"2026-02-20T22:24:57.880402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880604+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T22:24:57.880604+0000","last_peered":"2026-02-20T22:24:57.880604+0000","last_clean":"2026-02-20T22:24:57.880604+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T22:24:57.880604+0000","last_undegraded":"2026-02-20T22:24:57.880604+0000","last_fullsized":"2026-02-20T22:24:57.880604+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880714+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T22:24:57.880714+0000","last_peered":"2026-02-20T22:24:57.880714+0000","last_clean":"2026-02-20T22:24:57.880714+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T22:24:57.880714+0000","last_undegraded":"2026-02-20T22:24:57.880714+0000","last_fullsized":"2026-02-20T22:24:57.880714+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.881920+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T22:24:57.881920+0000","last_peered":"2026-02-20T22:24:57.881920+0000","last_clean":"2026-02-20T22:24:57.881920+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T22:24:57.881920+0000","last_undegraded":"2026-02-20T22:24:57.881920+0000","last_fullsized":"2026-02-20T22:24:57.881920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:58.151393+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T22:24:58.151393+0000","last_peered":"2026-02-20T22:24:58.151393+0000","last_clean":"2026-02-20T22:24:58.151393+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T22:24:58.151393+0000","last_undegraded":"2026-02-20T22:24:58.151393+0000","last_fullsized":"2026-02-20T22:24:58.151393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:58.151353+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T22:24:58.151353+0000","last_peered":"2026-02-20T22:24:58.151353+0000","last_clean":"2026-02-20T22:24:58.151353+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T22:24:58.151353+0000","last_undegraded":"2026-02-20T22:24:58.151353+0000","last_fullsized":"2026-02-20T22:24:58.151353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880363+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T22:24:57.880363+0000","last_peered":"2026-02-20T22:24:57.880363+0000","last_clean":"2026-02-20T22:24:57.880363+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T22:24:57.880363+0000","last_undegraded":"2026-02-20T22:24:57.880363+0000","last_fullsized":"2026-02-20T22:24:57.880363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":109,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.879920+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T22:24:57.879920+0000","last_peered":"2026-02-20T22:24:57.879920+0000","last_clean":"2026-02-20T22:24:57.879920+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T22:24:57.879920+0000","last_undegraded":"2026-02-20T22:24:57.879920+0000","last_fullsized":"2026-02-20T22:24:57.879920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":112,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":114688,"data_stored":31852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":720896,"data_stored":625803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":720896,"data_stored":625803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T22:25:00.267 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T22:25:00.426 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:00.426 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T22:25:00.436 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-20T22:24:58.809609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82400,"kb_used_data":1520,"kb_used_omap":23,"kb_used_meta":80808,"kb_avail":283033120,"statfs":{"total":289910292480,"available":289825914880,"internally_reserved":0,"allocated":1556480,"data_stored":1283458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23725,"internal_metadata":82748243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.926096"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880876+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T22:24:57.880876+0000","last_peered":"2026-02-20T22:24:57.880876+0000","last_clean":"2026-02-20T22:24:57.880876+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T22:24:57.880876+0000","last_undegraded":"2026-02-20T22:24:57.880876+0000","last_fullsized":"2026-02-20T22:24:57.880876+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880402+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T22:24:57.880402+0000","last_peered":"2026-02-20T22:24:57.880402+0000","last_clean":"2026-02-20T22:24:57.880402+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T22:24:57.880402+0000","last_undegraded":"2026-02-20T22:24:57.880402+0000","last_fullsized":"2026-02-20T22:24:57.880402+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880604+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T22:24:57.880604+0000","last_peered":"2026-02-20T22:24:57.880604+0000","last_clean":"2026-02-20T22:24:57.880604+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T22:24:57.880604+0000","last_undegraded":"2026-02-20T22:24:57.880604+0000","last_fullsized":"2026-02-20T22:24:57.880604+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880714+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T22:24:57.880714+0000","last_peered":"2026-02-20T22:24:57.880714+0000","last_clean":"2026-02-20T22:24:57.880714+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T22:24:57.880714+0000","last_undegraded":"2026-02-20T22:24:57.880714+0000","last_fullsized":"2026-02-20T22:24:57.880714+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.881920+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T22:24:57.881920+0000","last_peered":"2026-02-20T22:24:57.881920+0000","last_clean":"2026-02-20T22:24:57.881920+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T22:24:57.881920+0000","last_undegraded":"2026-02-20T22:24:57.881920+0000","last_fullsized":"2026-02-20T22:24:57.881920+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:58.151393+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T22:24:58.151393+0000","last_peered":"2026-02-20T22:24:58.151393+0000","last_clean":"2026-02-20T22:24:58.151393+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T22:24:58.151393+0000","last_undegraded":"2026-02-20T22:24:58.151393+0000","last_fullsized":"2026-02-20T22:24:58.151393+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:58.151353+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T22:24:58.151353+0000","last_peered":"2026-02-20T22:24:58.151353+0000","last_clean":"2026-02-20T22:24:58.151353+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T22:24:58.151353+0000","last_undegraded":"2026-02-20T22:24:58.151353+0000","last_fullsized":"2026-02-20T22:24:58.151353+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.880363+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T22:24:57.880363+0000","last_peered":"2026-02-20T22:24:57.880363+0000","last_clean":"2026-02-20T22:24:57.880363+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T22:24:57.880363+0000","last_undegraded":"2026-02-20T22:24:57.880363+0000","last_fullsized":"2026-02-20T22:24:57.880363+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":109,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:24:57.879920+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T22:24:57.879920+0000","last_peered":"2026-02-20T22:24:57.879920+0000","last_clean":"2026-02-20T22:24:57.879920+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T22:24:57.879920+0000","last_undegraded":"2026-02-20T22:24:57.879920+0000","last_fullsized":"2026-02-20T22:24:57.879920+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27072,"kb_used_data":112,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344768,"statfs":{"total":96636764160,"available":96609042432,"internally_reserved":0,"allocated":114688,"data_stored":31852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7475,"internal_metadata":27583181},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":720896,"data_stored":625803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":720896,"data_stored":625803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T22:25:00.438 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-20T22:25:00.438 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2026-02-20T22:25:00.438 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2026-02-20T22:25:00.438 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2026-02-20T22:25:00.613 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:00.613 INFO:teuthology.orchestra.run.trial032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-20T22:25:00.624 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2026-02-20T22:25:00.624 INFO:teuthology.run_tasks:Running task ceph... 2026-02-20T22:25:00.640 INFO:tasks.ceph:Creating ceph cluster cluster2... 2026-02-20T22:25:00.640 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2026-02-20T22:25:00.640 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587', 'branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'flavor': 'default', 'job_id': '61587', 'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'trial', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial', 'no_nested_subset': False, 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20260219.182737-tentacle'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, 'owner': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'priority': 69, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 7161, 'sha1': 'f7870454149f22574086dd2ff55d050585136531', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rbd', 'suite_branch': 'wip-pdonnell-testing-20260219.182737-tentacle', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_f7870454149f22574086dd2ff55d050585136531/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'f7870454149f22574086dd2ff55d050585136531', 'targets': {'trial032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDZKsPo+lvwLxn7Ee212rieC6zWJB+5QbSZDG9k7krFYPsVwEopXdvw5hQt2KBegIIEdhoyaRHkkH4ZdwY0vIO0=', 'trial107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOh8ioD937Osrcchl2HKEKdPFRr9QLyyACeqMcp5QXhMDbgV03LgBME8Q//ctKqZ6OIYcKsXBxkwhD+hdFR1J9c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'f7870454149f22574086dd2ff55d050585136531'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'fa17720d0088c3ac28e473468bfc79eeaff5cd38', 'timestamp': '2026-02-20_16:15:01', 'tube': 'trial', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.1540699'} 2026-02-20T22:25:00.641 DEBUG:teuthology.orchestra.run.trial032:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-20T22:25:00.642 DEBUG:teuthology.orchestra.run.trial107:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2026-02-20T22:25:00.646 DEBUG:teuthology.orchestra.run.trial032:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T22:25:00.690 DEBUG:teuthology.orchestra.run.trial107:> sudo install -d -m0777 -- /var/run/ceph 2026-02-20T22:25:00.699 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:00.700 DEBUG:teuthology.orchestra.run.trial107:> dd if=/scratch_devs of=/dev/stdout 2026-02-20T22:25:00.745 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2026-02-20T22:25:00.746 DEBUG:teuthology.orchestra.run.trial107:> stat /dev/vg_nvme/lv_1 2026-02-20T22:25:00.793 INFO:teuthology.orchestra.run.trial107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2026-02-20T22:25:00.793 INFO:teuthology.orchestra.run.trial107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout:Device: 5h/5d Inode: 897 Links: 1 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout:Access: 2026-02-20 22:23:48.039756634 +0000 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout:Modify: 2026-02-20 22:23:47.871759491 +0000 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout:Change: 2026-02-20 22:23:47.871759491 +0000 2026-02-20T22:25:00.794 INFO:teuthology.orchestra.run.trial107.stdout: Birth: - 2026-02-20T22:25:00.794 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2026-02-20T22:25:00.844 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records in 2026-02-20T22:25:00.844 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records out 2026-02-20T22:25:00.844 INFO:teuthology.orchestra.run.trial107.stderr:512 bytes copied, 0.000189702 s, 2.7 MB/s 2026-02-20T22:25:00.846 DEBUG:teuthology.orchestra.run.trial107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2026-02-20T22:25:00.895 DEBUG:teuthology.orchestra.run.trial107:> stat /dev/vg_nvme/lv_2 2026-02-20T22:25:00.941 INFO:teuthology.orchestra.run.trial107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2026-02-20T22:25:00.941 INFO:teuthology.orchestra.run.trial107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:25:00.941 INFO:teuthology.orchestra.run.trial107.stdout:Device: 5h/5d Inode: 920 Links: 1 2026-02-20T22:25:00.941 INFO:teuthology.orchestra.run.trial107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:25:00.942 INFO:teuthology.orchestra.run.trial107.stdout:Access: 2026-02-20 22:23:48.371750990 +0000 2026-02-20T22:25:00.942 INFO:teuthology.orchestra.run.trial107.stdout:Modify: 2026-02-20 22:23:48.191754051 +0000 2026-02-20T22:25:00.942 INFO:teuthology.orchestra.run.trial107.stdout:Change: 2026-02-20 22:23:48.191754051 +0000 2026-02-20T22:25:00.942 INFO:teuthology.orchestra.run.trial107.stdout: Birth: - 2026-02-20T22:25:00.942 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2026-02-20T22:25:00.992 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records in 2026-02-20T22:25:00.992 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records out 2026-02-20T22:25:00.993 INFO:teuthology.orchestra.run.trial107.stderr:512 bytes copied, 0.000176838 s, 2.9 MB/s 2026-02-20T22:25:00.994 DEBUG:teuthology.orchestra.run.trial107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2026-02-20T22:25:01.043 DEBUG:teuthology.orchestra.run.trial107:> stat /dev/vg_nvme/lv_3 2026-02-20T22:25:01.089 INFO:teuthology.orchestra.run.trial107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2026-02-20T22:25:01.089 INFO:teuthology.orchestra.run.trial107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:25:01.089 INFO:teuthology.orchestra.run.trial107.stdout:Device: 5h/5d Inode: 944 Links: 1 2026-02-20T22:25:01.089 INFO:teuthology.orchestra.run.trial107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:25:01.089 INFO:teuthology.orchestra.run.trial107.stdout:Access: 2026-02-20 22:23:48.699745413 +0000 2026-02-20T22:25:01.090 INFO:teuthology.orchestra.run.trial107.stdout:Modify: 2026-02-20 22:23:48.527748337 +0000 2026-02-20T22:25:01.090 INFO:teuthology.orchestra.run.trial107.stdout:Change: 2026-02-20 22:23:48.527748337 +0000 2026-02-20T22:25:01.090 INFO:teuthology.orchestra.run.trial107.stdout: Birth: - 2026-02-20T22:25:01.090 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2026-02-20T22:25:01.137 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records in 2026-02-20T22:25:01.137 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records out 2026-02-20T22:25:01.137 INFO:teuthology.orchestra.run.trial107.stderr:512 bytes copied, 0.00018887 s, 2.7 MB/s 2026-02-20T22:25:01.138 DEBUG:teuthology.orchestra.run.trial107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2026-02-20T22:25:01.182 DEBUG:teuthology.orchestra.run.trial107:> stat /dev/vg_nvme/lv_4 2026-02-20T22:25:01.229 INFO:teuthology.orchestra.run.trial107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2026-02-20T22:25:01.229 INFO:teuthology.orchestra.run.trial107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2026-02-20T22:25:01.229 INFO:teuthology.orchestra.run.trial107.stdout:Device: 5h/5d Inode: 969 Links: 1 2026-02-20T22:25:01.229 INFO:teuthology.orchestra.run.trial107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2026-02-20T22:25:01.229 INFO:teuthology.orchestra.run.trial107.stdout:Access: 2026-02-20 22:23:49.031739769 +0000 2026-02-20T22:25:01.230 INFO:teuthology.orchestra.run.trial107.stdout:Modify: 2026-02-20 22:23:48.847742897 +0000 2026-02-20T22:25:01.230 INFO:teuthology.orchestra.run.trial107.stdout:Change: 2026-02-20 22:23:48.847742897 +0000 2026-02-20T22:25:01.230 INFO:teuthology.orchestra.run.trial107.stdout: Birth: - 2026-02-20T22:25:01.230 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2026-02-20T22:25:01.276 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records in 2026-02-20T22:25:01.276 INFO:teuthology.orchestra.run.trial107.stderr:1+0 records out 2026-02-20T22:25:01.276 INFO:teuthology.orchestra.run.trial107.stderr:512 bytes copied, 0.000169615 s, 3.0 MB/s 2026-02-20T22:25:01.278 DEBUG:teuthology.orchestra.run.trial107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2026-02-20T22:25:01.327 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:25:01.327 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@trial107.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T22:25:01.327 INFO:tasks.ceph:Generating config... 2026-02-20T22:25:01.329 INFO:tasks.ceph:[client] debug rbd = 20 2026-02-20T22:25:01.329 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2026-02-20T22:25:01.329 INFO:tasks.ceph:[client] log to stderr = False 2026-02-20T22:25:01.329 INFO:tasks.ceph:[client] rbd default features = 125 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.330 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.330 INFO:tasks.ceph:[global] mon client directed command retry = 5 2026-02-20T22:25:01.330 INFO:tasks.ceph:[global] mon warn on pool no app = False 2026-02-20T22:25:01.331 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2026-02-20T22:25:01.331 INFO:tasks.ceph:[mgr] debug mgr = 20 2026-02-20T22:25:01.331 INFO:tasks.ceph:[mgr] debug ms = 1 2026-02-20T22:25:01.331 INFO:tasks.ceph:[mon] debug mon = 20 2026-02-20T22:25:01.331 INFO:tasks.ceph:[mon] debug ms = 1 2026-02-20T22:25:01.331 INFO:tasks.ceph:[mon] debug paxos = 20 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] bdev async discard = True 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] bdev enable discard = True 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2026-02-20T22:25:01.331 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] debug ms = 1 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] debug osd = 20 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2026-02-20T22:25:01.332 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2026-02-20T22:25:01.332 INFO:tasks.ceph:Setting up cluster2.mon.a... 2026-02-20T22:25:01.333 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.385 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.388 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.448 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.497 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '10.20.193.107')] 2026-02-20T22:25:01.498 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '10.20.193.107', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd mclock skip benchmark': 'true', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2026-02-20T22:25:01.499 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:01.499 DEBUG:teuthology.orchestra.run.trial107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T22:25:01.542 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 10.20.193.107 --print /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:monmaptool: generated fsid 22bff7d5-8e70-4f68-8060-d4046dcd49e4 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:setting min_mon_release = tentacle 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:epoch 0 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:fsid 22bff7d5-8e70-4f68-8060-d4046dcd49e4 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:last_changed 2026-02-20T22:25:01.600467+0000 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:created 2026-02-20T22:25:01.600467+0000 2026-02-20T22:25:01.602 INFO:teuthology.orchestra.run.trial107.stdout:min_mon_release 20 (tentacle) 2026-02-20T22:25:01.603 INFO:teuthology.orchestra.run.trial107.stdout:election_strategy: 1 2026-02-20T22:25:01.603 INFO:teuthology.orchestra.run.trial107.stdout:0: [v2:10.20.193.107:3300/0,v1:10.20.193.107:6789/0] mon.a 2026-02-20T22:25:01.603 INFO:teuthology.orchestra.run.trial107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2026-02-20T22:25:01.604 DEBUG:teuthology.orchestra.run.trial107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2026-02-20T22:25:01.648 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 22bff7d5-8e70-4f68-8060-d4046dcd49e4... 2026-02-20T22:25:01.649 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-20T22:25:01.650 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2026-02-20T22:25:01.691 INFO:teuthology.orchestra.run.trial032.stdout:[global] 2026-02-20T22:25:01.691 INFO:teuthology.orchestra.run.trial032.stdout: chdir = "" 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: auth supported = cephx 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: filestore xattr use omap = true 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: mon clock drift allowed = 1.000 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: osd crush chooseleaf type = 0 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: auth debug = true 2026-02-20T22:25:01.692 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: ms die on old message = true 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: ms die on bug = true 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: mon pg warn max object skew = 0 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: osd pool default size = 2 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.693 INFO:teuthology.orchestra.run.trial032.stdout: mon osd allow primary affinity = true 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon osd allow pg remap = true 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on legacy crush tunables = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on crush straw calc version zero = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on no sortbitwise = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on osd down out interval zero = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on too few osds = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: mon_allow_pool_size_one = true 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.694 INFO:teuthology.orchestra.run.trial032.stdout: osd default data pool replay window = 5 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon allow pool delete = true 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon cluster log file level = debug 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: debug asserts on shutdown = true 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon health detail to clog = false 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon host = 10.20.193.107 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon client directed command retry = 5 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: mon warn on pool no app = False 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: ms inject socket failures = 5000 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: fsid = 22bff7d5-8e70-4f68-8060-d4046dcd49e4 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout:[osd] 2026-02-20T22:25:01.695 INFO:teuthology.orchestra.run.trial032.stdout: osd journal size = 100 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd scrub load threshold = 5.0 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd scrub max interval = 600 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd mclock profile = high_recovery_ops 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd mclock skip benchmark = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd recover clone overlap = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd recovery max chunk = 1048576 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd debug shutdown = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd debug op order = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd debug verify stray on activate = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: osd debug trim objects = true 2026-02-20T22:25:01.696 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: osd open classes on start = true 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: osd debug pg log writeout = true 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: osd deep scrub update digest min age = 30 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: osd map max advance = 10 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: journal zero on create = true 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: filestore ondisk finisher threads = 3 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: filestore apply finisher threads = 3 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: bdev debug aio = true 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: osd debug misdirected ops = true 2026-02-20T22:25:01.697 INFO:teuthology.orchestra.run.trial032.stdout: bdev async discard = True 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: bdev enable discard = True 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: bluestore allocator = stupid 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: bluestore block size = 96636764160 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: bluestore fsck on mount = True 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: debug bluefs = 1/20 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: debug bluestore = 1/20 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: debug osd = 20 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: debug rocksdb = 4/10 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: mon osd backfillfull_ratio = 0.85 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: mon osd full ratio = 0.9 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: mon osd nearfull ratio = 0.8 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: osd failsafe full ratio = 0.95 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: osd objectstore = bluestore 2026-02-20T22:25:01.698 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout:[mgr] 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug mgr = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug mon = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug auth = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: mgr/telemetry/nag = false 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout:[mon] 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug ms = 1 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug mon = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug paxos = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: debug auth = 20 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: mon data avail warn = 5 2026-02-20T22:25:01.699 INFO:teuthology.orchestra.run.trial032.stdout: mon mgr mkfs grace = 240 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon osd reporter subtree level = osd 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon osd prime pg temp = true 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: # 1m isn't quite enough 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: mon_down_mkfs_grace = 2m 2026-02-20T22:25:01.700 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: mon_warn_on_filestore_osds = false 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout:[client] 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: rgw cache enabled = true 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: rgw enable ops log = true 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: rgw enable usage log = true 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: debug rbd = 20 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: debug rbd_mirror = 15 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: log to stderr = False 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: rbd default features = 125 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: rbd mirror image policy type = simple 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout:[mon.a] 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.0] 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.701 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.1] 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.2] 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout:[client.mirror.3] 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.702 INFO:teuthology.orchestra.run.trial032.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.709 INFO:teuthology.orchestra.run.trial107.stdout:[global] 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: chdir = "" 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: auth supported = cephx 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: filestore xattr use omap = true 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: mon clock drift allowed = 1.000 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: osd crush chooseleaf type = 0 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: auth debug = true 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: ms die on old message = true 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: ms die on bug = true 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: mon max pg per osd = 10000 # >= luminous 2026-02-20T22:25:01.710 INFO:teuthology.orchestra.run.trial107.stdout: mon pg warn max object skew = 0 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: # disable pg_autoscaler by default for new pools 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: osd_pool_default_pg_autoscale_mode = off 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: osd pool default size = 2 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon osd allow primary affinity = true 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon osd allow pg remap = true 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on legacy crush tunables = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on crush straw calc version zero = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on no sortbitwise = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on osd down out interval zero = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on too few osds = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_pool_no_redundancy = false 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: mon_allow_pool_size_one = true 2026-02-20T22:25:01.711 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: osd pool default erasure code profile = plugin=isa technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: osd default data pool replay window = 5 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon allow pool delete = true 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon cluster log file level = debug 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: debug asserts on shutdown = true 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon health detail to clog = false 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon host = 10.20.193.107 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon client directed command retry = 5 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: mon warn on pool no app = False 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: ms inject socket failures = 5000 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: fsid = 22bff7d5-8e70-4f68-8060-d4046dcd49e4 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout:[osd] 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: osd journal size = 100 2026-02-20T22:25:01.712 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd scrub load threshold = 5.0 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd scrub max interval = 600 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd mclock profile = high_recovery_ops 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd mclock skip benchmark = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd recover clone overlap = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd recovery max chunk = 1048576 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd debug shutdown = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd debug op order = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd debug verify stray on activate = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd debug trim objects = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd open classes on start = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd debug pg log writeout = true 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.713 INFO:teuthology.orchestra.run.trial107.stdout: osd deep scrub update digest min age = 30 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: osd map max advance = 10 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: journal zero on create = true 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: filestore ondisk finisher threads = 3 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: filestore apply finisher threads = 3 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bdev debug aio = true 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: osd debug misdirected ops = true 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bdev async discard = True 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bdev enable discard = True 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bluestore allocator = stupid 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bluestore block size = 96636764160 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: bluestore fsck on mount = True 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: debug bluefs = 1/20 2026-02-20T22:25:01.714 INFO:teuthology.orchestra.run.trial107.stdout: debug bluestore = 1/20 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug osd = 20 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug rocksdb = 4/10 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mon osd backfillfull_ratio = 0.85 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mon osd full ratio = 0.9 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mon osd nearfull ratio = 0.8 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: osd failsafe full ratio = 0.95 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: osd objectstore = bluestore 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout:[mgr] 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug mgr = 20 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug mon = 20 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: debug auth = 20 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: mgr/telemetry/nag = false 2026-02-20T22:25:01.715 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout:[mon] 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: debug ms = 1 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: debug mon = 20 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: debug paxos = 20 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: debug auth = 20 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon data avail warn = 5 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon mgr mkfs grace = 240 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min pgs per osd = 4 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon osd reporter subtree level = osd 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon osd prime pg temp = true 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: mon reweight min bytes per osd = 10 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: # rotate auth tickets quickly to exercise renewal paths 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: auth mon ticket ttl = 660 # 11m 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: auth service ticket ttl = 240 # 4m 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.716 INFO:teuthology.orchestra.run.trial107.stdout: # don't complain about insecure global_id in the test suite 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: # 1m isn't quite enough 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: mon_down_mkfs_grace = 2m 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: mon_warn_on_filestore_osds = false 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout:[client] 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: rgw cache enabled = true 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: rgw enable ops log = true 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: rgw enable usage log = true 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: debug rbd = 20 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: debug rbd_mirror = 15 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: log to stderr = False 2026-02-20T22:25:01.717 INFO:teuthology.orchestra.run.trial107.stdout: rbd default features = 125 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: rbd mirror image policy type = simple 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout:[mon.a] 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.0] 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.1] 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.2] 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.3] 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2026-02-20T22:25:01.718 INFO:teuthology.orchestra.run.trial107.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2026-02-20T22:25:01.719 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2026-02-20T22:25:01.720 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.784 INFO:tasks.ceph:Copying monmap to all nodes... 2026-02-20T22:25:01.784 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:01.784 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2026-02-20T22:25:01.830 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:01.830 DEBUG:teuthology.orchestra.run.trial107:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2026-02-20T22:25:01.877 INFO:tasks.ceph:Sending monmap to node ubuntu@trial032.front.sepia.ceph.com 2026-02-20T22:25:01.878 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:25:01.878 DEBUG:teuthology.orchestra.run.trial032:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-20T22:25:01.878 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.893 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:25:01.893 DEBUG:teuthology.orchestra.run.trial032:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:01.941 INFO:tasks.ceph:Sending monmap to node ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:25:01.941 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:01.941 DEBUG:teuthology.orchestra.run.trial107:> sudo dd of=/etc/ceph/cluster2.keyring 2026-02-20T22:25:01.941 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2026-02-20T22:25:01.954 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:01.955 DEBUG:teuthology.orchestra.run.trial107:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:02.001 INFO:tasks.ceph:Setting up mon nodes... 2026-02-20T22:25:02.002 INFO:tasks.ceph:Setting up mgr nodes... 2026-02-20T22:25:02.002 DEBUG:teuthology.orchestra.run.trial032:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-20T22:25:02.025 INFO:teuthology.orchestra.run.trial032.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2026-02-20T22:25:02.028 INFO:tasks.ceph:Setting up mds nodes... 2026-02-20T22:25:02.028 INFO:tasks.ceph_client:Setting up client nodes... 2026-02-20T22:25:02.028 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2026-02-20T22:25:02.089 INFO:teuthology.orchestra.run.trial032.stdout:creating /etc/ceph/cluster2.client.0.keyring 2026-02-20T22:25:02.096 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2026-02-20T22:25:02.115 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2026-02-20T22:25:02.123 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-20T22:25:02.182 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2026-02-20T22:25:02.189 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-20T22:25:02.250 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2026-02-20T22:25:02.257 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-20T22:25:02.314 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2026-02-20T22:25:02.321 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-20T22:25:02.378 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2026-02-20T22:25:02.385 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-20T22:25:02.442 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2026-02-20T22:25:02.449 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-20T22:25:02.506 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2026-02-20T22:25:02.513 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-20T22:25:02.575 INFO:teuthology.orchestra.run.trial107.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2026-02-20T22:25:02.583 INFO:tasks.ceph:Running mkfs on osd nodes... 2026-02-20T22:25:02.583 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@trial032.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@trial107.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2026-02-20T22:25:02.584 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2026-02-20T22:25:02.633 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:25:02.633 INFO:tasks.ceph:role: cluster2.osd.0 2026-02-20T22:25:02.633 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:25:02.634 DEBUG:teuthology.orchestra.run.trial107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2026-02-20T22:25:02.682 INFO:teuthology.orchestra.run.trial107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:25:02.682 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:25:02.683 INFO:teuthology.orchestra.run.trial107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:25:03.269 INFO:teuthology.orchestra.run.trial107.stdout:Discarding blocks...Done. 2026-02-20T22:25:03.271 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@trial107.front.sepia.ceph.com -o noatime 2026-02-20T22:25:03.271 DEBUG:teuthology.orchestra.run.trial107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2026-02-20T22:25:03.325 DEBUG:teuthology.orchestra.run.trial107:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2026-02-20T22:25:03.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:25:03.380 INFO:teuthology.orchestra.run.trial107.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:25:03.380 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2026-02-20T22:25:03.429 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:25:03.429 INFO:tasks.ceph:role: cluster2.osd.1 2026-02-20T22:25:03.429 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:25:03.429 DEBUG:teuthology.orchestra.run.trial107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2026-02-20T22:25:03.479 INFO:teuthology.orchestra.run.trial107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:25:03.480 INFO:teuthology.orchestra.run.trial107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:25:04.046 INFO:teuthology.orchestra.run.trial107.stdout:Discarding blocks...Done. 2026-02-20T22:25:04.047 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@trial107.front.sepia.ceph.com -o noatime 2026-02-20T22:25:04.047 DEBUG:teuthology.orchestra.run.trial107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2026-02-20T22:25:04.103 DEBUG:teuthology.orchestra.run.trial107:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2026-02-20T22:25:04.150 INFO:teuthology.orchestra.run.trial107.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:25:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:25:04.151 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2026-02-20T22:25:04.204 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2026-02-20T22:25:04.204 INFO:tasks.ceph:role: cluster2.osd.2 2026-02-20T22:25:04.205 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:25:04.205 DEBUG:teuthology.orchestra.run.trial107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=45875200 blks 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 attr=2, projid32bit=1 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout: = reflink=1 bigtime=0 inobtcount=0 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout:data = bsize=4096 blocks=183500800, imaxpct=25 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout: = sunit=0 swidth=0 blks 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout:log =internal log bsize=4096 blocks=89600, version=2 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout: = sectsz=4096 sunit=1 blks, lazy-count=1 2026-02-20T22:25:04.254 INFO:teuthology.orchestra.run.trial107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2026-02-20T22:25:04.828 INFO:teuthology.orchestra.run.trial107.stdout:Discarding blocks...Done. 2026-02-20T22:25:04.829 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@trial107.front.sepia.ceph.com -o noatime 2026-02-20T22:25:04.830 DEBUG:teuthology.orchestra.run.trial107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2026-02-20T22:25:04.890 DEBUG:teuthology.orchestra.run.trial107:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2026-02-20T22:25:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:25:04.940 INFO:teuthology.orchestra.run.trial107.stderr:sudo: /sbin/restorecon: command not found 2026-02-20T22:25:04.940 DEBUG:teuthology.orchestra.run.trial107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:05.006 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:05.002+0000 7fd0066baa40 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2026-02-20T22:25:05.007 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:05.002+0000 7fd0066baa40 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2026-02-20T22:25:05.007 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:05.002+0000 7fd0066baa40 -1 bdev(0x559f59aaf800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2026-02-20T22:25:05.007 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:05.002+0000 7fd0066baa40 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2026-02-20T22:25:05.973 DEBUG:teuthology.orchestra.run.trial107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2026-02-20T22:25:06.026 DEBUG:teuthology.orchestra.run.trial107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:06.090 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:06.086+0000 7fbc7768ca40 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2026-02-20T22:25:06.090 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:06.086+0000 7fbc7768ca40 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2026-02-20T22:25:06.090 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:06.086+0000 7fbc7768ca40 -1 bdev(0x56483dc89800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2026-02-20T22:25:06.091 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:06.086+0000 7fbc7768ca40 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2026-02-20T22:25:07.018 DEBUG:teuthology.orchestra.run.trial107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2026-02-20T22:25:07.070 DEBUG:teuthology.orchestra.run.trial107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:07.134 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:07.130+0000 7fd1e69c6a40 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2026-02-20T22:25:07.135 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:07.130+0000 7fd1e69c6a40 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2026-02-20T22:25:07.135 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:07.130+0000 7fd1e69c6a40 -1 bdev(0x563a35dc3800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2026-02-20T22:25:07.135 INFO:teuthology.orchestra.run.trial107.stderr:2026-02-20T22:25:07.130+0000 7fd1e69c6a40 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2026-02-20T22:25:08.033 DEBUG:teuthology.orchestra.run.trial107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2026-02-20T22:25:08.082 INFO:tasks.ceph:Reading keys from all nodes... 2026-02-20T22:25:08.082 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:25:08.082 DEBUG:teuthology.orchestra.run.trial032:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2026-02-20T22:25:08.094 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.094 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2026-02-20T22:25:08.134 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.134 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2026-02-20T22:25:08.186 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.186 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2026-02-20T22:25:08.238 DEBUG:teuthology.orchestra.run.trial032:> set -ex 2026-02-20T22:25:08.238 DEBUG:teuthology.orchestra.run.trial032:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2026-02-20T22:25:08.244 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.244 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2026-02-20T22:25:08.285 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.285 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2026-02-20T22:25:08.329 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.330 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2026-02-20T22:25:08.379 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.379 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2026-02-20T22:25:08.425 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.425 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2026-02-20T22:25:08.469 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.469 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2026-02-20T22:25:08.513 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.513 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2026-02-20T22:25:08.557 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:08.557 DEBUG:teuthology.orchestra.run.trial107:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2026-02-20T22:25:08.601 INFO:tasks.ceph:Adding keys to all mons... 2026-02-20T22:25:08.602 DEBUG:teuthology.orchestra.run.trial107:> sudo tee -a /etc/ceph/cluster2.keyring 2026-02-20T22:25:08.648 INFO:teuthology.orchestra.run.trial107.stdout:[mgr.x] 2026-02-20T22:25:08.648 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpUiZ0ARAAjZIJ1VPQTeqq74pcLeZ3qQ== 2026-02-20T22:25:08.648 INFO:teuthology.orchestra.run.trial107.stdout:[osd.0] 2026-02-20T22:25:08.648 INFO:teuthology.orchestra.run.trial107.stdout: key = AQBB35hp7idWABAAJnOShGDH7/n/nIu1uCA1GA== 2026-02-20T22:25:08.648 INFO:teuthology.orchestra.run.trial107.stdout:[osd.1] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQBC35hpyctIBRAADepMg1IwLHGp0Kfuv/tROw== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[osd.2] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQBD35hpUmLsBxAAs3av1yQQ9AjL0seCL6cHNg== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[client.0] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpf9I5BRAASAURqJbQJ0qyJKKa/c5YVQ== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpmsnRBhAAHbJttcoChZp6Vr4Vr6Ht0A== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.0] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hp6+THChAAKqcpjCf6f6xiE0E4Rs1shQ== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.1] 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpINbUDhAApVtRURJ058UCD9Pyt8RPHQ== 2026-02-20T22:25:08.649 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.2] 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpjZCkEhAA6/WcjfpesP1uGTdwqMIFTw== 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.3] 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpphZyFhAATtzVQefxOhvibqiIyPFG8Q== 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.4] 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hp4NJDGhAAiUCa8qwebhZiZEf3FAgxqQ== 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.5] 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpaW4WHhAAauc+6UtzzwsTTOVEfSxJmg== 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.6] 2026-02-20T22:25:08.650 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpglI8IhAAcF+fx6AjaYXgSeXLiX3TSQ== 2026-02-20T22:25:08.651 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2026-02-20T22:25:08.715 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:25:08.778 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:25:08.842 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2026-02-20T22:25:08.906 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:08.971 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.035 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.099 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.164 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.227 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.291 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.355 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.419 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2026-02-20T22:25:09.484 INFO:tasks.ceph:Running mkfs on mon nodes... 2026-02-20T22:25:09.484 DEBUG:teuthology.orchestra.run.trial107:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2026-02-20T22:25:09.533 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2026-02-20T22:25:09.603 DEBUG:teuthology.orchestra.run.trial107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2026-02-20T22:25:09.654 DEBUG:teuthology.orchestra.run.trial107:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2026-02-20T22:25:09.701 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2026-02-20T22:25:09.701 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2026-02-20T22:25:09.701 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2026-02-20T22:25:09.743 INFO:tasks.ceph.cluster2.mon.a:Started 2026-02-20T22:25:09.743 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2026-02-20T22:25:09.743 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2026-02-20T22:25:09.744 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2026-02-20T22:25:09.746 INFO:tasks.ceph.cluster2.mgr.x:Started 2026-02-20T22:25:09.746 DEBUG:tasks.ceph:set 0 configs 2026-02-20T22:25:09.746 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 config dump 2026-02-20T22:25:09.868 INFO:teuthology.orchestra.run.trial107.stdout:WHO MASK LEVEL OPTION VALUE RO 2026-02-20T22:25:09.878 INFO:tasks.ceph:Setting crush tunables to default 2026-02-20T22:25:09.879 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 osd crush tunables default 2026-02-20T22:25:09.975 INFO:teuthology.orchestra.run.trial107.stderr:adjusted tunables profile to default 2026-02-20T22:25:09.984 INFO:tasks.ceph:check_enable_crimson: False 2026-02-20T22:25:09.984 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2026-02-20T22:25:09.985 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:09.985 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2026-02-20T22:25:09.995 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:09.995 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2026-02-20T22:25:10.046 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:10.046 DEBUG:teuthology.orchestra.run.trial107:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2026-02-20T22:25:10.098 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 osd new 67b9c177-c488-4711-9869-3c6812b373b9 0 2026-02-20T22:25:10.239 INFO:teuthology.orchestra.run.trial107.stdout:0 2026-02-20T22:25:10.249 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 osd new 4ca2fdff-829f-4705-91d4-bf1aa309394c 1 2026-02-20T22:25:10.344 INFO:teuthology.orchestra.run.trial107.stdout:1 2026-02-20T22:25:10.354 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 osd new b2eeafe2-8f74-402f-9490-aa4cb0904997 2 2026-02-20T22:25:10.450 INFO:teuthology.orchestra.run.trial107.stdout:2 2026-02-20T22:25:10.459 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2026-02-20T22:25:10.460 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2026-02-20T22:25:10.462 INFO:tasks.ceph.cluster2.osd.0:Started 2026-02-20T22:25:10.462 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2026-02-20T22:25:10.462 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2026-02-20T22:25:10.464 INFO:tasks.ceph.cluster2.osd.1:Started 2026-02-20T22:25:10.464 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2026-02-20T22:25:10.464 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2026-02-20T22:25:10.466 INFO:tasks.ceph.cluster2.osd.2:Started 2026-02-20T22:25:10.467 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-20T22:25:10.559 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:10.560 INFO:teuthology.orchestra.run.trial107.stdout:{"epoch":5,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T22:25:10.448417+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:25:10.571 INFO:tasks.ceph.ceph_manager.cluster2:[] 2026-02-20T22:25:10.572 INFO:tasks.ceph:Waiting for OSDs to come up 2026-02-20T22:25:10.874 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-20T22:25:10.971 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:25:10.966+0000 7f72fc60ca40 -1 Falling back to public interface 2026-02-20T22:25:10.974 INFO:teuthology.misc.health.trial107.stdout: 2026-02-20T22:25:10.975 INFO:teuthology.misc.health.trial107.stdout:{"epoch":5,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T22:25:10.448417+0000","last_up_change":"0.000000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:25:10.980 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:ERROR:root:Module 'jmespath' is not installed. 2026-02-20T22:25:10.983 DEBUG:teuthology.misc:0 of 3 OSDs are up 2026-02-20T22:25:11.018 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:ERROR:dashboard.controllers.nvmeof:Failed to import NVMeoFClient and related components: cannot import name 'NVMeoFClient' from 'dashboard.services.nvmeof_client' (/usr/share/ceph/mgr/dashboard/services/nvmeof_client.py) 2026-02-20T22:25:11.019 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:ERROR:sso:Module 'jmespath' is not installed. 2026-02-20T22:25:11.032 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:ERROR:root:Module 'xmltodict' is not installed. 2026-02-20T22:25:11.038 INFO:tasks.ceph.cluster2.osd.1.trial107.stderr:2026-02-20T22:25:11.034+0000 7fc7d74e9a40 -1 Falling back to public interface 2026-02-20T22:25:11.174 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:25:11.170+0000 7f7ee3ab7a40 -1 Falling back to public interface 2026-02-20T22:25:11.381 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2026-02-20T22:25:11.382 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2026-02-20T22:25:11.382 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr: from numpy import show_config as show_numpy_config 2026-02-20T22:25:12.121 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:25:12.114+0000 7f72fc60ca40 -1 osd.0 0 log_to_monitors true 2026-02-20T22:25:12.173 INFO:tasks.ceph.cluster2.osd.1.trial107.stderr:2026-02-20T22:25:12.166+0000 7fc7d74e9a40 -1 osd.1 0 log_to_monitors true 2026-02-20T22:25:12.324 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:25:12.318+0000 7f7ee3ab7a40 -1 osd.2 0 log_to_monitors true 2026-02-20T22:25:12.442 INFO:tasks.ceph.cluster2.mgr.x.trial032.stderr:2026-02-20T22:25:12.438+0000 7efe32851640 -1 mgr.server handle_report got status from non-daemon mon.a 2026-02-20T22:25:13.853 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:25:13.850+0000 7f72f7d92640 -1 osd.0 0 waiting for initial osdmap 2026-02-20T22:25:13.854 INFO:tasks.ceph.cluster2.osd.1.trial107.stderr:2026-02-20T22:25:13.850+0000 7fc7d2c6f640 -1 osd.1 0 waiting for initial osdmap 2026-02-20T22:25:13.856 INFO:tasks.ceph.cluster2.osd.1.trial107.stderr:2026-02-20T22:25:13.850+0000 7fc7caa88640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:25:13.856 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:25:13.850+0000 7f72efbab640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:25:13.861 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:25:13.858+0000 7f7edf23d640 -1 osd.2 0 waiting for initial osdmap 2026-02-20T22:25:13.864 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:25:13.858+0000 7f7ed7056640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2026-02-20T22:25:17.286 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2026-02-20T22:25:17.444 INFO:teuthology.misc.health.trial107.stdout: 2026-02-20T22:25:17.444 INFO:teuthology.misc.health.trial107.stdout:{"epoch":10,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T22:25:16.837882+0000","last_up_change":"2026-02-20T22:25:14.831864+0000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:25:15.471458+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6800","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6801","nonce":1374819874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6802","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6803","nonce":1374819874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6806","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6807","nonce":1374819874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6804","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6805","nonce":1374819874}]},"public_addr":"10.20.193.107:6801/1374819874","cluster_addr":"10.20.193.107:6803/1374819874","heartbeat_back_addr":"10.20.193.107:6807/1374819874","heartbeat_front_addr":"10.20.193.107:6805/1374819874","state":["exists","up"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6808","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6809","nonce":1913959616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6810","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6811","nonce":1913959616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6814","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6815","nonce":1913959616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6812","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6813","nonce":1913959616}]},"public_addr":"10.20.193.107:6809/1913959616","cluster_addr":"10.20.193.107:6811/1913959616","heartbeat_back_addr":"10.20.193.107:6815/1913959616","heartbeat_front_addr":"10.20.193.107:6813/1913959616","state":["exists","up"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6816","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6817","nonce":2773181681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6818","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6819","nonce":2773181681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6822","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6823","nonce":2773181681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6820","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6821","nonce":2773181681}]},"public_addr":"10.20.193.107:6817/2773181681","cluster_addr":"10.20.193.107:6819/2773181681","heartbeat_back_addr":"10.20.193.107:6823/2773181681","heartbeat_front_addr":"10.20.193.107:6821/2773181681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:25:17.455 DEBUG:teuthology.misc:3 of 3 OSDs are up 2026-02-20T22:25:17.456 INFO:tasks.ceph:Creating RBD pool 2026-02-20T22:25:17.456 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 osd pool create rbd 8 2026-02-20T22:25:17.844 INFO:teuthology.orchestra.run.trial107.stderr:pool 'rbd' created 2026-02-20T22:25:17.859 DEBUG:teuthology.orchestra.run.trial107:> rbd --cluster cluster2 pool init rbd 2026-02-20T22:25:20.859 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2026-02-20T22:25:20.860 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 config log 1 --format=json 2026-02-20T22:25:20.861 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2026-02-20T22:25:21.009 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:21.019 INFO:teuthology.orchestra.run.trial107.stdout:[{"version":1,"timestamp":"0.000000","name":"","changes":[]}] 2026-02-20T22:25:21.019 INFO:tasks.ceph_manager:config epoch is 1 2026-02-20T22:25:21.019 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2026-02-20T22:25:21.019 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2026-02-20T22:25:21.020 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2026-02-20T22:25:21.191 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:21.201 INFO:teuthology.orchestra.run.trial107.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6826","nonce":3919710898},{"type":"v1","addr":"10.20.193.32:6827","nonce":3919710898}]},"active_addr":"10.20.193.32:6827/3919710898","active_change":"2026-02-20T22:25:11.432268+0000","active_mgr_features":4544132024016699391,"available":true,"standbys":[],"modules":["iostat","nfs"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_pg_upmap_activity":{"name":"update_pg_upmap_activity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Updates pg_upmap activity stats to be used in `balancer status detail`","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"certificate_automated_rotation_enabled":{"name":"certificate_automated_rotation_enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"This flag controls whether cephadm automatically rotates certificates upon expiration.","long_desc":"","tags":[],"see_also":[]},"certificate_check_debug_mode":{"name":"certificate_check_debug_mode","type":"bool","level":"dev","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"FOR TESTING ONLY: This flag forces the certificate check instead of waiting for certificate_check_period.","long_desc":"","tags":[],"see_also":[]},"certificate_check_period":{"name":"certificate_check_period","type":"int","level":"advanced","flags":0,"default_value":"1","min":"0","max":"30","enum_allowed":[],"desc":"Specifies how often (in days) the certificate should be checked for validity.","long_desc":"","tags":[],"see_also":[]},"certificate_duration_days":{"name":"certificate_duration_days","type":"int","level":"advanced","flags":0,"default_value":"1095","min":"90","max":"3650","enum_allowed":[],"desc":"Specifies the duration of self certificates generated and signed by cephadm root CA","long_desc":"","tags":[],"see_also":[]},"certificate_renewal_threshold_days":{"name":"certificate_renewal_threshold_days","type":"int","level":"advanced","flags":0,"default_value":"30","min":"10","max":"90","enum_allowed":[],"desc":"Specifies the lead time in days to initiate certificate renewal before expiration.","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.28.1","min":"","max":"","enum_allowed":[],"desc":"Alertmanager container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"Elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:12.3.1","min":"","max":"","enum_allowed":[],"desc":"Grafana container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"Haproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.9.1","min":"","max":"","enum_allowed":[],"desc":"Node exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.5","min":"","max":"","enum_allowed":[],"desc":"Nvmeof container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"Oauth2 proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v3.6.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:ceph20-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba metrics container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"Snmp gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"quay.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"stray_daemon_check_interval":{"name":"stray_daemon_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"how frequently cephadm should check for the presence of stray daemons","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROM_ALERT_CREDENTIAL_CACHE_TTL":{"name":"PROM_ALERT_CREDENTIAL_CACHE_TTL","type":"int","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_HOSTNAME_PER_DAEMON":{"name":"RGW_HOSTNAME_PER_DAEMON","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crypto_caller":{"name":"crypto_caller","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sso_oauth2":{"name":"sso_oauth2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_cloning":{"name":"pause_cloning","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"pause_purging":{"name":"pause_purging","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Pause asynchronous subvolume purge threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"tentacle":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"force_disabled_modules":{},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":3153508977}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":1782535183}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":283771405}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"10.20.193.32:0","nonce":3966795412}]}]} 2026-02-20T22:25:21.204 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2026-02-20T22:25:21.204 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2026-02-20T22:25:21.204 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-20T22:25:21.366 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:21.367 INFO:teuthology.orchestra.run.trial107.stdout:{"epoch":14,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T22:25:20.846485+0000","last_up_change":"2026-02-20T22:25:14.831864+0000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:25:15.471458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:25:17.609108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6800","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6801","nonce":1374819874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6802","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6803","nonce":1374819874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6806","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6807","nonce":1374819874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6804","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6805","nonce":1374819874}]},"public_addr":"10.20.193.107:6801/1374819874","cluster_addr":"10.20.193.107:6803/1374819874","heartbeat_back_addr":"10.20.193.107:6807/1374819874","heartbeat_front_addr":"10.20.193.107:6805/1374819874","state":["exists","up"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6808","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6809","nonce":1913959616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6810","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6811","nonce":1913959616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6814","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6815","nonce":1913959616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6812","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6813","nonce":1913959616}]},"public_addr":"10.20.193.107:6809/1913959616","cluster_addr":"10.20.193.107:6811/1913959616","heartbeat_back_addr":"10.20.193.107:6815/1913959616","heartbeat_front_addr":"10.20.193.107:6813/1913959616","state":["exists","up"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6816","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6817","nonce":2773181681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6818","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6819","nonce":2773181681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6822","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6823","nonce":2773181681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6820","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6821","nonce":2773181681}]},"public_addr":"10.20.193.107:6817/2773181681","cluster_addr":"10.20.193.107:6819/2773181681","heartbeat_back_addr":"10.20.193.107:6823/2773181681","heartbeat_front_addr":"10.20.193.107:6821/2773181681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:25:13.130051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:25:21.377 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2026-02-20T22:25:21.378 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-20T22:25:21.535 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:21.535 INFO:teuthology.orchestra.run.trial107.stdout:{"epoch":14,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T22:25:20.846485+0000","last_up_change":"2026-02-20T22:25:14.831864+0000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:25:15.471458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:25:17.609108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6800","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6801","nonce":1374819874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6802","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6803","nonce":1374819874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6806","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6807","nonce":1374819874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6804","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6805","nonce":1374819874}]},"public_addr":"10.20.193.107:6801/1374819874","cluster_addr":"10.20.193.107:6803/1374819874","heartbeat_back_addr":"10.20.193.107:6807/1374819874","heartbeat_front_addr":"10.20.193.107:6805/1374819874","state":["exists","up"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6808","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6809","nonce":1913959616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6810","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6811","nonce":1913959616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6814","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6815","nonce":1913959616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6812","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6813","nonce":1913959616}]},"public_addr":"10.20.193.107:6809/1913959616","cluster_addr":"10.20.193.107:6811/1913959616","heartbeat_back_addr":"10.20.193.107:6815/1913959616","heartbeat_front_addr":"10.20.193.107:6813/1913959616","state":["exists","up"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6816","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6817","nonce":2773181681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6818","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6819","nonce":2773181681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6822","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6823","nonce":2773181681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6820","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6821","nonce":2773181681}]},"public_addr":"10.20.193.107:6817/2773181681","cluster_addr":"10.20.193.107:6819/2773181681","heartbeat_back_addr":"10.20.193.107:6823/2773181681","heartbeat_front_addr":"10.20.193.107:6821/2773181681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:25:13.130051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T22:25:21.546 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2026-02-20T22:25:21.546 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2026-02-20T22:25:21.547 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2026-02-20T22:25:21.630 INFO:teuthology.orchestra.run.trial107.stdout:34359738371 2026-02-20T22:25:21.631 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-20T22:25:21.632 INFO:teuthology.orchestra.run.trial107.stdout:34359738371 2026-02-20T22:25:21.633 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-20T22:25:21.634 INFO:teuthology.orchestra.run.trial107.stdout:34359738372 2026-02-20T22:25:21.634 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-20T22:25:21.793 INFO:teuthology.orchestra.run.trial107.stdout:34359738369 2026-02-20T22:25:21.805 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738369 for osd.0 2026-02-20T22:25:21.826 INFO:teuthology.orchestra.run.trial107.stdout:34359738369 2026-02-20T22:25:21.832 INFO:teuthology.orchestra.run.trial107.stdout:34359738370 2026-02-20T22:25:21.839 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738369 for osd.2 2026-02-20T22:25:21.844 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738370 for osd.1 2026-02-20T22:25:22.806 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-20T22:25:22.840 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-20T22:25:22.845 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-20T22:25:22.965 INFO:teuthology.orchestra.run.trial107.stdout:34359738370 2026-02-20T22:25:22.977 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.0 2026-02-20T22:25:22.998 INFO:teuthology.orchestra.run.trial107.stdout:34359738371 2026-02-20T22:25:23.000 INFO:teuthology.orchestra.run.trial107.stdout:34359738370 2026-02-20T22:25:23.010 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738371 for osd.1 2026-02-20T22:25:23.013 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738370 for osd.2 2026-02-20T22:25:23.978 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2026-02-20T22:25:24.011 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2026-02-20T22:25:24.013 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2026-02-20T22:25:24.143 INFO:teuthology.orchestra.run.trial107.stdout:34359738371 2026-02-20T22:25:24.155 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.0 2026-02-20T22:25:24.156 DEBUG:teuthology.parallel:result is None 2026-02-20T22:25:24.173 INFO:teuthology.orchestra.run.trial107.stdout:34359738371 2026-02-20T22:25:24.175 INFO:teuthology.orchestra.run.trial107.stdout:34359738372 2026-02-20T22:25:24.184 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738371 got 34359738371 for osd.2 2026-02-20T22:25:24.185 DEBUG:teuthology.parallel:result is None 2026-02-20T22:25:24.188 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738372 got 34359738372 for osd.1 2026-02-20T22:25:24.188 DEBUG:teuthology.parallel:result is None 2026-02-20T22:25:24.188 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-20T22:25:24.189 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T22:25:24.380 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:24.381 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T22:25:24.390 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-20T22:25:23.436982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82784,"kb_used_data":1528,"kb_used_omap":21,"kb_used_meta":80810,"kb_avail":283032736,"statfs":{"total":289910292480,"available":289825521664,"internally_reserved":0,"allocated":1564672,"data_stored":1287172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21775,"internal_metadata":82750193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.589859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853571+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T22:25:20.853571+0000","last_peered":"2026-02-20T22:25:20.853571+0000","last_clean":"2026-02-20T22:25:20.853571+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T22:25:20.853571+0000","last_undegraded":"2026-02-20T22:25:20.853571+0000","last_fullsized":"2026-02-20T22:25:20.853571+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853388+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T22:25:20.853388+0000","last_peered":"2026-02-20T22:25:20.853388+0000","last_clean":"2026-02-20T22:25:20.853388+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T22:25:20.853388+0000","last_undegraded":"2026-02-20T22:25:20.853388+0000","last_fullsized":"2026-02-20T22:25:20.853388+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853351+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T22:25:20.853351+0000","last_peered":"2026-02-20T22:25:20.853351+0000","last_clean":"2026-02-20T22:25:20.853351+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T22:25:20.853351+0000","last_undegraded":"2026-02-20T22:25:20.853351+0000","last_fullsized":"2026-02-20T22:25:20.853351+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.852956+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T22:25:20.852956+0000","last_peered":"2026-02-20T22:25:20.852956+0000","last_clean":"2026-02-20T22:25:20.852956+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T22:25:20.852956+0000","last_undegraded":"2026-02-20T22:25:20.852956+0000","last_fullsized":"2026-02-20T22:25:20.852956+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.851616+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T22:25:20.851616+0000","last_peered":"2026-02-20T22:25:20.851616+0000","last_clean":"2026-02-20T22:25:20.851616+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T22:25:20.851616+0000","last_undegraded":"2026-02-20T22:25:20.851616+0000","last_fullsized":"2026-02-20T22:25:20.851616+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.848430+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T22:25:20.848430+0000","last_peered":"2026-02-20T22:25:20.848430+0000","last_clean":"2026-02-20T22:25:20.848430+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T22:25:20.848430+0000","last_undegraded":"2026-02-20T22:25:20.848430+0000","last_fullsized":"2026-02-20T22:25:20.848430+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.848372+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T22:25:20.848372+0000","last_peered":"2026-02-20T22:25:20.848372+0000","last_clean":"2026-02-20T22:25:20.848372+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T22:25:20.848372+0000","last_undegraded":"2026-02-20T22:25:20.848372+0000","last_fullsized":"2026-02-20T22:25:20.848372+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853000+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T22:25:20.853000+0000","last_peered":"2026-02-20T22:25:20.853000+0000","last_clean":"2026-02-20T22:25:20.853000+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T22:25:20.853000+0000","last_undegraded":"2026-02-20T22:25:20.853000+0000","last_fullsized":"2026-02-20T22:25:20.853000+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":109,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.852623+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T22:25:20.852623+0000","last_peered":"2026-02-20T22:25:20.852623+0000","last_clean":"2026-02-20T22:25:20.852623+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T22:25:20.852623+0000","last_undegraded":"2026-02-20T22:25:20.852623+0000","last_fullsized":"2026-02-20T22:25:20.852623+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":720896,"data_stored":625853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":120,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":122880,"data_stored":35466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5525,"internal_metadata":27585131},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":720896,"data_stored":625853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T22:25:24.392 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T22:25:24.541 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:24.542 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T22:25:24.553 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2026-02-20T22:25:23.436982+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82784,"kb_used_data":1528,"kb_used_omap":21,"kb_used_meta":80810,"kb_avail":283032736,"statfs":{"total":289910292480,"available":289825521664,"internally_reserved":0,"allocated":1564672,"data_stored":1287172,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21775,"internal_metadata":82750193},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.589859"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853571+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T22:25:20.853571+0000","last_peered":"2026-02-20T22:25:20.853571+0000","last_clean":"2026-02-20T22:25:20.853571+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T22:25:20.853571+0000","last_undegraded":"2026-02-20T22:25:20.853571+0000","last_fullsized":"2026-02-20T22:25:20.853571+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853388+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T22:25:20.853388+0000","last_peered":"2026-02-20T22:25:20.853388+0000","last_clean":"2026-02-20T22:25:20.853388+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T22:25:20.853388+0000","last_undegraded":"2026-02-20T22:25:20.853388+0000","last_fullsized":"2026-02-20T22:25:20.853388+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853351+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T22:25:20.853351+0000","last_peered":"2026-02-20T22:25:20.853351+0000","last_clean":"2026-02-20T22:25:20.853351+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T22:25:20.853351+0000","last_undegraded":"2026-02-20T22:25:20.853351+0000","last_fullsized":"2026-02-20T22:25:20.853351+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.852956+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T22:25:20.852956+0000","last_peered":"2026-02-20T22:25:20.852956+0000","last_clean":"2026-02-20T22:25:20.852956+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T22:25:20.852956+0000","last_undegraded":"2026-02-20T22:25:20.852956+0000","last_fullsized":"2026-02-20T22:25:20.852956+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"14'2","reported_seq":22,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.851616+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T22:25:20.851616+0000","last_peered":"2026-02-20T22:25:20.851616+0000","last_clean":"2026-02-20T22:25:20.851616+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T22:25:20.851616+0000","last_undegraded":"2026-02-20T22:25:20.851616+0000","last_fullsized":"2026-02-20T22:25:20.851616+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.848430+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T22:25:20.848430+0000","last_peered":"2026-02-20T22:25:20.848430+0000","last_clean":"2026-02-20T22:25:20.848430+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T22:25:20.848430+0000","last_undegraded":"2026-02-20T22:25:20.848430+0000","last_fullsized":"2026-02-20T22:25:20.848430+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.848372+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T22:25:20.848372+0000","last_peered":"2026-02-20T22:25:20.848372+0000","last_clean":"2026-02-20T22:25:20.848372+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T22:25:20.848372+0000","last_undegraded":"2026-02-20T22:25:20.848372+0000","last_fullsized":"2026-02-20T22:25:20.848372+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"12'1","reported_seq":21,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.853000+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T22:25:20.853000+0000","last_peered":"2026-02-20T22:25:20.853000+0000","last_clean":"2026-02-20T22:25:20.853000+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T22:25:20.853000+0000","last_undegraded":"2026-02-20T22:25:20.853000+0000","last_fullsized":"2026-02-20T22:25:20.853000+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'72","reported_seq":109,"reported_epoch":14,"state":"active+clean","last_fresh":"2026-02-20T22:25:20.852623+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T22:25:20.852623+0000","last_peered":"2026-02-20T22:25:20.852623+0000","last_clean":"2026-02-20T22:25:20.852623+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T22:25:20.852623+0000","last_undegraded":"2026-02-20T22:25:20.852623+0000","last_fullsized":"2026-02-20T22:25:20.852623+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":720896,"data_stored":625853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738371,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27080,"kb_used_data":120,"kb_used_omap":5,"kb_used_meta":26938,"kb_avail":94344760,"statfs":{"total":96636764160,"available":96609034240,"internally_reserved":0,"allocated":122880,"data_stored":35466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5525,"internal_metadata":27585131},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27852,"kb_used_data":704,"kb_used_omap":7,"kb_used_meta":26936,"kb_avail":94343988,"statfs":{"total":96636764160,"available":96608243712,"internally_reserved":0,"allocated":720896,"data_stored":625853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8125,"internal_metadata":27582531},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T22:25:24.554 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-20T22:25:24.554 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2026-02-20T22:25:24.555 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2026-02-20T22:25:24.555 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2026-02-20T22:25:24.732 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T22:25:24.732 INFO:teuthology.orchestra.run.trial107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2026-02-20T22:25:24.743 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2026-02-20T22:25:24.743 INFO:teuthology.run_tasks:Running task exec... 2026-02-20T22:25:24.761 INFO:teuthology.task.exec:Executing custom commands... 2026-02-20T22:25:24.761 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@trial107.front.sepia.ceph.com 2026-02-20T22:25:24.762 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:24.952 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror] 2026-02-20T22:25:24.952 INFO:teuthology.orchestra.run.trial107.stdout: key = AQAn35hpaCRiBxAAJZFxT5FUm/nncx38BKBFVw== 2026-02-20T22:25:24.952 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-20T22:25:24.952 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:24.952 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror 2026-02-20T22:25:24.963 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:25.151 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.0] 2026-02-20T22:25:25.151 INFO:teuthology.orchestra.run.trial107.stdout: key = AQAn35hp8d3iCxAAgEh2Zh1h0n1PjWXqVLxVrA== 2026-02-20T22:25:25.151 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:25.151 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:25.152 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.0 2026-02-20T22:25:25.164 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:25.354 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.1] 2026-02-20T22:25:25.354 INFO:teuthology.orchestra.run.trial107.stdout: key = AQAn35hpuZUkEBAAz/mv65oyNEOsWR4ynt0wnA== 2026-02-20T22:25:25.354 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:25.354 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:25.355 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.1 2026-02-20T22:25:25.364 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:25.551 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.2] 2026-02-20T22:25:25.551 INFO:teuthology.orchestra.run.trial107.stdout: key = AQAn35hpIF9sFBAA4wQvsv+eUTA7KtXjHpaVkQ== 2026-02-20T22:25:25.551 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:25.551 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:25.551 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.2 2026-02-20T22:25:25.562 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:25.746 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.3] 2026-02-20T22:25:25.747 INFO:teuthology.orchestra.run.trial107.stdout: key = AQAn35hpj++yGBAA4vHEQChCCthgYPQ8cMM1eA== 2026-02-20T22:25:25.747 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:25.747 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:25.747 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.3 2026-02-20T22:25:25.759 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:25.946 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror] 2026-02-20T22:25:25.946 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpmsnRBhAAHbJttcoChZp6Vr4Vr6Ht0A== 2026-02-20T22:25:25.946 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror-peer" 2026-02-20T22:25:25.946 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:25.946 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror 2026-02-20T22:25:25.959 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:26.143 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.0] 2026-02-20T22:25:26.143 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hp6+THChAAKqcpjCf6f6xiE0E4Rs1shQ== 2026-02-20T22:25:26.144 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:26.144 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:26.144 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.0 2026-02-20T22:25:26.157 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:26.344 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.1] 2026-02-20T22:25:26.344 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpINbUDhAApVtRURJ058UCD9Pyt8RPHQ== 2026-02-20T22:25:26.345 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:26.345 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:26.345 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.1 2026-02-20T22:25:26.355 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:26.542 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.2] 2026-02-20T22:25:26.543 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpjZCkEhAA6/WcjfpesP1uGTdwqMIFTw== 2026-02-20T22:25:26.543 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:26.543 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:26.543 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.2 2026-02-20T22:25:26.554 DEBUG:teuthology.orchestra.run.trial107:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2026-02-20T22:25:26.742 INFO:teuthology.orchestra.run.trial107.stdout:[client.mirror.3] 2026-02-20T22:25:26.742 INFO:teuthology.orchestra.run.trial107.stdout: key = AQA+35hpphZyFhAATtzVQefxOhvibqiIyPFG8Q== 2026-02-20T22:25:26.742 INFO:teuthology.orchestra.run.trial107.stdout: caps mon = "profile rbd-mirror" 2026-02-20T22:25:26.742 INFO:teuthology.orchestra.run.trial107.stdout: caps osd = "profile rbd" 2026-02-20T22:25:26.742 INFO:teuthology.orchestra.run.trial107.stderr:updated caps for client.mirror.3 2026-02-20T22:25:26.753 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.774 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-20T22:25:26.774 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-20T22:25:26.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-20T22:25:26.775 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.786 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-20T22:25:26.786 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-20T22:25:26.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-20T22:25:26.787 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.804 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-20T22:25:26.804 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-20T22:25:26.806 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-20T22:25:26.806 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:25:26.822 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:25:26.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:25:26.824 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.840 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:25:26.841 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:25:26.843 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:25:26.843 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-20T22:25:26.906 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-20T22:25:26.907 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-20T22:25:26.907 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.918 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-20T22:25:26.919 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-20T22:25:26.920 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-20T22:25:26.920 INFO:teuthology.run_tasks:Running task rbd-mirror... 2026-02-20T22:25:26.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-20T22:25:26.932 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-20T22:25:26.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-20T22:25:26.933 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2026-02-20T22:25:26.947 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1771626326 2026-02-20T22:25:26.948 DEBUG:tasks.rbd_mirror_thrash:Yielding 2026-02-20T22:25:26.948 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2026-02-20T22:25:26.959 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1771626326 2026-02-20T22:25:26.959 DEBUG:tasks.rbd_mirror_thrash:Yielding 2026-02-20T22:25:26.959 INFO:teuthology.run_tasks:Running task workunit... 2026-02-20T22:25:26.981 INFO:tasks.workunit:Pulling workunits from ref f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:25:26.982 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-20T22:25:26.982 DEBUG:teuthology.orchestra.run.trial107:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-20T22:25:26.983 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2026-02-20T22:25:26.983 INFO:tasks.rbd_mirror_thrash:waiting for 100.32163519123057 secs before thrashing 2026-02-20T22:25:26.983 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2026-02-20T22:25:26.983 INFO:tasks.rbd_mirror_thrash:waiting for 118.45364980472789 secs before thrashing 2026-02-20T22:25:26.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T22:25:26.987 INFO:teuthology.orchestra.run.trial107.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2026-02-20T22:25:26.987 DEBUG:teuthology.orchestra.run.trial107:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-20T22:25:27.032 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-20T22:25:27.033 DEBUG:teuthology.orchestra.run.trial107:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2026-02-20T22:25:27.077 INFO:tasks.workunit:timeout=6h 2026-02-20T22:25:27.077 INFO:tasks.workunit:cleanup=True 2026-02-20T22:25:27.078 DEBUG:teuthology.orchestra.run.trial107:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout f7870454149f22574086dd2ff55d050585136531 2026-02-20T22:25:27.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Note: switching to 'f7870454149f22574086dd2ff55d050585136531'. 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:state without impacting any branches by switching back to a branch. 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: git switch -c 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Or undo this operation with: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: git switch - 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:25:48.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:HEAD is now at f7870454149 ceph-dev-pipeline: configure 2026-02-20T22:25:48.806 DEBUG:teuthology.orchestra.run.trial107:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2026-02-20T22:25:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2026-02-20T22:25:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-20T22:25:48.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2026-02-20T22:25:48.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2026-02-20T22:25:48.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2026-02-20T22:25:49.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2026-02-20T22:25:49.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-20T22:25:49.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2026-02-20T22:25:49.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2026-02-20T22:25:49.040 DEBUG:teuthology.orchestra.run.trial107:> set -ex 2026-02-20T22:25:49.040 DEBUG:teuthology.orchestra.run.trial107:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2026-02-20T22:25:49.092 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2026-02-20T22:25:49.094 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2026-02-20T22:25:49.095 DEBUG:teuthology.orchestra.run.trial107:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f7870454149f22574086dd2ff55d050585136531 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-20T22:25:49.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ IMAGE_COUNT=50 2026-02-20T22:25:49.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ export LOCKDEP=0 2026-02-20T22:25:49.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ LOCKDEP=0 2026-02-20T22:25:49.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-20T22:25:49.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ RBD_MIRROR_INSTANCES=4 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CLUSTER1=cluster1 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CLUSTER2=cluster2 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ PEER_CLUSTER_SUFFIX= 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ POOL=mirror 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ PARENT_POOL=mirror_parent 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ NS1=ns1 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ NS2=ns2 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ TEMPDIR= 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ID=mirror 2026-02-20T22:25:49.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ RBD_MIRROR_MODE=snapshot 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ MIRROR_POOL_MODE=image 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' snapshot = snapshot ']' 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ MIRROR_POOL_MODE=image 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ export 'CEPH_ARGS=--id mirror' 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS='--id mirror' 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ LAST_MIRROR_INSTANCE=3 2026-02-20T22:25:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2026-02-20T22:25:49.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_BIN=. 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_SRC=. 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' -e CMakeCache.txt ']' 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 -gt 0 ']' 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ setup 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local c 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ trap 'cleanup $?' INT TERM EXIT 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ setup_tempdir 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mktemp -d 2026-02-20T22:25:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ TEMPDIR=/tmp/tmp.l7jgOKLHDi 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z 1 ']' 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ setup_pools cluster1 cluster2 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster2 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mon_map_file 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mon_addr 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local admin_key_file 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:49.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2026-02-20T22:25:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror' already exists 2026-02-20T22:25:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2026-02-20T22:25:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror_parent' already exists 2026-02-20T22:25:50.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:50.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 pool init mirror 2026-02-20T22:25:53.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:53.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2026-02-20T22:25:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:56.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2026-02-20T22:25:56.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2026-02-20T22:25:56.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2026-02-20T22:25:56.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2026-02-20T22:25:57.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 namespace create mirror_parent/ns1 2026-02-20T22:25:57.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2026-02-20T22:25:57.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2026-02-20T22:25:57.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent/ns1 image 2026-02-20T22:25:57.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z ']' 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z ']' 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_add cluster1 mirror cluster2 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local client_cluster=cluster2 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster2 2026-02-20T22:25:57.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid_var_name 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local error_code 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local peer_uuid 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 16 32 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:25:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2026-02-20T22:25:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_uuid=9c3eb251-6775-4590-86ea-1896e78dca23 2026-02-20T22:25:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T22:25:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:25:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 17 ']' 2026-02-20T22:25:57.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 -eq 0 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_add cluster1 mirror_parent cluster2 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror_parent 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local client_cluster=cluster2 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster2 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid_var_name 2026-02-20T22:25:57.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local error_code 2026-02-20T22:25:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local peer_uuid 2026-02-20T22:25:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 16 32 2026-02-20T22:25:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:25:57.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_uuid=d95fcab0-fbad-494f-a0b5-ac4b5d9c9412 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 17 ']' 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 -eq 0 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ setup_pools cluster2 cluster1 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster1 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mon_map_file 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mon_addr 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local admin_key_file 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2026-02-20T22:25:57.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror' already exists 2026-02-20T22:25:57.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:57.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2026-02-20T22:25:58.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror_parent' already exists 2026-02-20T22:25:58.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:25:58.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 pool init mirror 2026-02-20T22:26:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:26:01.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2026-02-20T22:26:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T22:26:04.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2026-02-20T22:26:04.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2026-02-20T22:26:04.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2026-02-20T22:26:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2026-02-20T22:26:04.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 namespace create mirror_parent/ns1 2026-02-20T22:26:04.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2026-02-20T22:26:04.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2026-02-20T22:26:04.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent/ns1 image 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z ']' 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z ']' 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_add cluster2 mirror cluster1 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:04.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local client_cluster=cluster1 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster1 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid_var_name 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local error_code 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local peer_uuid 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 16 32 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:26:04.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_uuid=6a8a824a-25d7-4b5f-8753-90d84e0d7a77 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 17 ']' 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 -eq 0 2026-02-20T22:26:04.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_add cluster2 mirror_parent cluster1 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror_parent 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local client_cluster=cluster1 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_cluster=cluster1 2026-02-20T22:26:04.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local uuid_var_name 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local error_code 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local peer_uuid 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 16 32 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:26:04.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2026-02-20T22:26:04.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ peer_uuid=9629f0bc-9843-46c6-b64a-392b5cee041e 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 17 ']' 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 -eq 0 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:04.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirrors cluster1 2026-02-20T22:26:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 0 3 2026-02-20T22:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster1:0 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:0 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster1:0 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster1 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=0 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster1 2026-02-20T22:26:04.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=0 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster1:1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster1:1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=1 2026-02-20T22:26:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster1 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=1 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster1:2 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:2 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster1:2 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster1 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=2 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster1 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=2 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster1:3 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:3 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster1:3 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster1 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=3 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster1 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=3 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirrors cluster2 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 0 3 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster2:0 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:0 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster2:0 cluster instance 2026-02-20T22:26:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster2:0 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster2 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=0 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 = cluster2:0 ']' 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster2 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster2 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=0 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster2:1 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:1 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster2:1 cluster instance 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster2:1 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster2 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=1 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 1 = cluster2:1 ']' 2026-02-20T22:26:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=1 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster2:2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster2:2 cluster instance 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster2:2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster2 2026-02-20T22:26:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=2 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 2 = cluster2:2 ']' 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster2 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster2 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=2 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster2:3 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:3 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster2:3 cluster instance 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster2:3 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:26:04.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster2 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=3 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 3 = cluster2:3 ']' 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster2 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster2 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=3 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:26:04 TEST: add image and test replay after client crashes 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:26:04 TEST: add image and test replay after client crashes 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=test 2026-02-20T22:26:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror test 512M 2026-02-20T22:26:04.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:26:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2026-02-20T22:26:04.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror test snapshot 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:26:04.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2026-02-20T22:26:05.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:26:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/test 2026-02-20T22:26:05.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:26:05.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:26:05.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:26:05.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:26:05.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10b6b65ac45c 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10b6b65ac45c 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:26:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:04 2026 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:26:04 2026 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:04 2026 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:26:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:26:05.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:26:05.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:05.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:05.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:05.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:26:05.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:05.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:05.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:26:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:26:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:05.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:26:06.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:06.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:06.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:06.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:06.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:06.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:06.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:06.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:06.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:06.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:06.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-20T22:26:06.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:06.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:26:06.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:26:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-20T22:26:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:06.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:06.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:06.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:26:06.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:26:06.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:26:06.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:26:06.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:06.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:06.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:06.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-20T22:26:06.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:06.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:06.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:06.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:06.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:26:06.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:26:06.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:06.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:06.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:06.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-20T22:26:06.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:06.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:06.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:06.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 1 2026-02-20T22:26:07.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:07.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:07.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:07.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:07.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:07.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:07.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:07.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-20T22:26:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:26:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:07.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:07.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:07.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:26:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:26:07.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:26:07.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-20T22:26:08.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:08.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:08.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:08.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:26:08.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:26:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:26:08.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:26:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:08.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:08.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:08.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-20T22:26:08.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:08.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:08.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:26:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:08.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:26:08.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:26:08.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:26:08.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-20T22:26:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:08.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:08.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 2 2026-02-20T22:26:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:10.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:10.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:10.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:10.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:10.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:10.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:10.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:10.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:26:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:26:10.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:26:10.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:10.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:10.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:10.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-20T22:26:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:10.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:26:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:26:10.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:26:10.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:10.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:10.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:10.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-20T22:26:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:10.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:26:10.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:10.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:26:10.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:26:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:10.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:10.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:10.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-20T22:26:10.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:10.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:10.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:10.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:10.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 4 2026-02-20T22:26:14.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:14.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:14.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:14.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:14.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:14.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:14.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:14.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-20T22:26:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:26:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:14.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:26:14.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:14.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:26:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:26:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:26:14.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:14.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-20T22:26:14.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:14.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:14.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:26:14.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:26:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:26:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:14.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:14.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-20T22:26:14.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:14.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:14.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:14.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:14.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-20T22:26:14.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:14.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:14.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:14.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:26:14.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:26:14.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:26:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:14.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-20T22:26:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:14.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:14.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:14.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 8 2026-02-20T22:26:22.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:22.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:22.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:22.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:22.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:22.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:22.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:22.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:22.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:22.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:22.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:22.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:22.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:22.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:26:22.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:26:22.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:26:22.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:22.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:22.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:26:22.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2026-02-20T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:22.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:26:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:26:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:26:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:26:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:26:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:26:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2026-02-20T22:26:22.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:22.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:22.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:22.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:26:22.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:26:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:26:22.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:26:22.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:26:22.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2026-02-20T22:26:22.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/test'] 2026-02-20T22:26:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:26:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:26:22.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:22.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 8 2026-02-20T22:26:31.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:26:31.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:26:31.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:26:31.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:26:31.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:26:31.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:26:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:26:31.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:26:31.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:26:31.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:26:31.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:26:31.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name= 2026-02-20T22:26:31.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 10 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:31.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:26:31.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:26:31.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:26:31.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:26:31.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 604160 bytes 2026-02-20T22:26:31.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 63550464 bytes 2026-02-20T22:26:31.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 140293120 bytes 2026-02-20T22:26:31.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 169553920 bytes 2026-02-20T22:26:31.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 247727104 bytes 2026-02-20T22:26:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 258855936 bytes 2026-02-20T22:26:31.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 298265600 bytes 2026-02-20T22:26:31.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 307645440 bytes 2026-02-20T22:26:31.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 338813952 bytes 2026-02-20T22:26:31.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 426144768 bytes 2026-02-20T22:26:31.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 437483520 bytes 2026-02-20T22:26:31.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 476252160 bytes 2026-02-20T22:26:31.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 519388160 bytes 2026-02-20T22:26:31.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 575002624 bytes 2026-02-20T22:26:31.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 656576512 bytes 2026-02-20T22:26:31.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 715857920 bytes 2026-02-20T22:26:31.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 751222784 bytes 2026-02-20T22:26:31.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 805772288 bytes 2026-02-20T22:26:31.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 828018688 bytes 2026-02-20T22:26:31.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 886084608 bytes 2026-02-20T22:26:31.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 911579136 bytes 2026-02-20T22:26:31.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 947279872 bytes 2026-02-20T22:26:31.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 959887360 bytes 2026-02-20T22:26:32.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1052007424 bytes 2026-02-20T22:26:32.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1087729664 bytes 2026-02-20T22:26:32.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1101853696 bytes 2026-02-20T22:26:32.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1119859712 bytes 2026-02-20T22:26:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1145603072 bytes 2026-02-20T22:26:32.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1185657856 bytes 2026-02-20T22:26:32.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1199036416 bytes 2026-02-20T22:26:32.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1241996288 bytes 2026-02-20T22:26:32.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1279050752 bytes 2026-02-20T22:26:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1310371840 bytes 2026-02-20T22:26:32.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1359576064 bytes 2026-02-20T22:26:32.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1389298688 bytes 2026-02-20T22:26:32.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1418909696 bytes 2026-02-20T22:26:32.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1440812032 bytes 2026-02-20T22:26:32.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1552355328 bytes 2026-02-20T22:26:32.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1594061824 bytes 2026-02-20T22:26:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1616906240 bytes 2026-02-20T22:26:32.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1640268800 bytes 2026-02-20T22:26:32.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1640870912 bytes 2026-02-20T22:26:32.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1704060928 bytes 2026-02-20T22:26:32.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1729808384 bytes 2026-02-20T22:26:32.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1764366336 bytes 2026-02-20T22:26:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1787981824 bytes 2026-02-20T22:26:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1813741568 bytes 2026-02-20T22:26:32.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1849921536 bytes 2026-02-20T22:26:32.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1967277056 bytes 2026-02-20T22:26:32.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1969608704 bytes 2026-02-20T22:26:32.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2016249856 bytes 2026-02-20T22:26:32.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2028025856 bytes 2026-02-20T22:26:33.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2073453568 bytes 2026-02-20T22:26:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2139813888 bytes 2026-02-20T22:26:33.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2201315328 bytes 2026-02-20T22:26:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2219327488 bytes 2026-02-20T22:26:33.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2257515520 bytes 2026-02-20T22:26:33.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2297637888 bytes 2026-02-20T22:26:33.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2299671552 bytes 2026-02-20T22:26:33.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2350129152 bytes 2026-02-20T22:26:33.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2425836544 bytes 2026-02-20T22:26:33.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2472694784 bytes 2026-02-20T22:26:33.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2500411392 bytes 2026-02-20T22:26:33.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2558900224 bytes 2026-02-20T22:26:33.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2595585024 bytes 2026-02-20T22:26:33.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2647151616 bytes 2026-02-20T22:26:33.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2661288960 bytes 2026-02-20T22:26:33.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2661909504 bytes 2026-02-20T22:26:33.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2696035328 bytes 2026-02-20T22:26:33.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2700812288 bytes 2026-02-20T22:26:33.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2731126784 bytes 2026-02-20T22:26:33.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2755989504 bytes 2026-02-20T22:26:33.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2846667776 bytes 2026-02-20T22:26:33.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2868976640 bytes 2026-02-20T22:26:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2913866752 bytes 2026-02-20T22:26:33.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2973496320 bytes 2026-02-20T22:26:33.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2992537600 bytes 2026-02-20T22:26:33.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3007181824 bytes 2026-02-20T22:26:33.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3110571008 bytes 2026-02-20T22:26:34.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3155826688 bytes 2026-02-20T22:26:34.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3187299328 bytes 2026-02-20T22:26:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3249864704 bytes 2026-02-20T22:26:34.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3287271424 bytes 2026-02-20T22:26:34.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3336849408 bytes 2026-02-20T22:26:34.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3418831872 bytes 2026-02-20T22:26:34.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3458320384 bytes 2026-02-20T22:26:34.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3491001344 bytes 2026-02-20T22:26:34.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3512294400 bytes 2026-02-20T22:26:34.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3519639552 bytes 2026-02-20T22:26:34.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3612289024 bytes 2026-02-20T22:26:34.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3696320512 bytes 2026-02-20T22:26:34.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3737337856 bytes 2026-02-20T22:26:34.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3852295168 bytes 2026-02-20T22:26:34.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3896370176 bytes 2026-02-20T22:26:34.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3961068544 bytes 2026-02-20T22:26:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4024937472 bytes 2026-02-20T22:26:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4076764160 bytes 2026-02-20T22:26:34.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4080054272 bytes 2026-02-20T22:26:34.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4105778176 bytes 2026-02-20T22:26:34.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4186254336 bytes 2026-02-20T22:26:34.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4213064704 bytes 2026-02-20T22:26:34.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4247342080 bytes 2026-02-20T22:26:35.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4278925312 bytes 2026-02-20T22:26:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4354247680 bytes 2026-02-20T22:26:35.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4369606656 bytes 2026-02-20T22:26:35.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4400442368 bytes 2026-02-20T22:26:35.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4428744704 bytes 2026-02-20T22:26:35.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4455735296 bytes 2026-02-20T22:26:35.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4505425920 bytes 2026-02-20T22:26:35.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4532398080 bytes 2026-02-20T22:26:35.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4623374336 bytes 2026-02-20T22:26:35.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4623995904 bytes 2026-02-20T22:26:35.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4680665088 bytes 2026-02-20T22:26:35.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4726614016 bytes 2026-02-20T22:26:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4792558592 bytes 2026-02-20T22:26:35.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4882780160 bytes 2026-02-20T22:26:35.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4917010432 bytes 2026-02-20T22:26:35.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4954497024 bytes 2026-02-20T22:26:35.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5020568576 bytes 2026-02-20T22:26:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5059471360 bytes 2026-02-20T22:26:35.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5092614144 bytes 2026-02-20T22:26:35.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5132137472 bytes 2026-02-20T22:26:35.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5157914624 bytes 2026-02-20T22:26:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5189446656 bytes 2026-02-20T22:26:35.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5235215360 bytes 2026-02-20T22:26:35.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5293566976 bytes 2026-02-20T22:26:35.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5302438912 bytes 2026-02-20T22:26:36.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5333683200 bytes 2026-02-20T22:26:36.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5334297600 bytes 2026-02-20T22:26:36.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5402403840 bytes 2026-02-20T22:26:36.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5523169280 bytes 2026-02-20T22:26:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5540927488 bytes 2026-02-20T22:26:36.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5595420672 bytes 2026-02-20T22:26:36.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5651345408 bytes 2026-02-20T22:26:36.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5678872576 bytes 2026-02-20T22:26:36.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5788934144 bytes 2026-02-20T22:26:36.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5858053120 bytes 2026-02-20T22:26:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5884491776 bytes 2026-02-20T22:26:36.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5951822848 bytes 2026-02-20T22:26:36.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6019860480 bytes 2026-02-20T22:26:36.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6041698304 bytes 2026-02-20T22:26:36.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6099447808 bytes 2026-02-20T22:26:36.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6152441856 bytes 2026-02-20T22:26:36.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6193946624 bytes 2026-02-20T22:26:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6273823744 bytes 2026-02-20T22:26:36.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6354678784 bytes 2026-02-20T22:26:36.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6412558336 bytes 2026-02-20T22:26:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6482861056 bytes 2026-02-20T22:26:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6509992960 bytes 2026-02-20T22:26:37.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6565931008 bytes 2026-02-20T22:26:37.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6644163584 bytes 2026-02-20T22:26:37.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6675639296 bytes 2026-02-20T22:26:37.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6722312192 bytes 2026-02-20T22:26:37.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6743866368 bytes 2026-02-20T22:26:37.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6819213312 bytes 2026-02-20T22:26:37.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6857659392 bytes 2026-02-20T22:26:37.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6905030656 bytes 2026-02-20T22:26:37.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6973494272 bytes 2026-02-20T22:26:37.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7019309056 bytes 2026-02-20T22:26:37.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7041867776 bytes 2026-02-20T22:26:37.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7063616512 bytes 2026-02-20T22:26:37.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7097065472 bytes 2026-02-20T22:26:37.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7127584768 bytes 2026-02-20T22:26:37.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7214217216 bytes 2026-02-20T22:26:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7232732160 bytes 2026-02-20T22:26:37.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7244931072 bytes 2026-02-20T22:26:37.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7269160960 bytes 2026-02-20T22:26:37.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7289787392 bytes 2026-02-20T22:26:37.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7312006144 bytes 2026-02-20T22:26:37.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7338432512 bytes 2026-02-20T22:26:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7381565440 bytes 2026-02-20T22:26:37.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7407012864 bytes 2026-02-20T22:26:37.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7468646400 bytes 2026-02-20T22:26:37.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7490049024 bytes 2026-02-20T22:26:37.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7509505024 bytes 2026-02-20T22:26:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7527429120 bytes 2026-02-20T22:26:37.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7539665920 bytes 2026-02-20T22:26:38.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7564582912 bytes 2026-02-20T22:26:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7617782784 bytes 2026-02-20T22:26:38.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7632854016 bytes 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:38.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:26:38.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:26:38.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:26:38.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:26:38.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:26:39.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:26:39.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:26:39.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:26:39.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771626365,"replay_state":"idle"} 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:26:29 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: down+unknown 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: status not found 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: ' 2026-02-20T22:26:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:26:39.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:26:39.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:26:39.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771626365,"replay_state":"idle"} 2026-02-20T22:26:39.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:26:39.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:26:39.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:26:39.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:26:39.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:26:39.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:26:39.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:26:39.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:26:29' 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:26:29' 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:26:39.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:26:29' 2026-02-20T22:26:39.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:26:39.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:26:39.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:26:39.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='WARNINGOKWARNING142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK42204268mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"remote_snapshot_timestamp":1771626365,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:26:29cluster2c3ececfc-6db9-428b-b666-3a05930e41b3down+unknownstatus not found' 2026-02-20T22:26:39.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:26:39.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:26:39.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:26:29' 2026-02-20T22:26:39.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:26:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:26:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:26:39.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:26:39.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:26:39.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:39.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:26:39.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap1 2026-02-20T22:26:39.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap1 2026-02-20T22:26:39.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:26:39.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:26:39.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:26:39.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1 2026-02-20T22:26:39.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:27:07.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6636837698248371) > weight (0.25) 2026-02-20T22:27:07.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3940943188690179) > weight (0.25) 2026-02-20T22:27:07.306 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4226143220831362) > weight (0.25) 2026-02-20T22:27:07.306 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-20T22:27:07.307 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-20T22:27:07.307 INFO:tasks.rbd_mirror_thrash:waiting for 2.045490597315105 secs before reviving daemons 2026-02-20T22:27:07.308 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T22:27:07.300+0000 7f7426ed4640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 19577) UID: 1000 2026-02-20T22:27:09.353 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-20T22:27:09.354 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:27:09.354 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-20T22:27:09.354 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:27:09.354 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:27:09.357 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:27:09.357 INFO:tasks.rbd_mirror_thrash:waiting for 103.20689703029697 secs before thrashing 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:27:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:27:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:27:09.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:27:10.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:27:10.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:27:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:27:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:27:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:27:10.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:27:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:27:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:27:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:27:10.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:27:10.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:27:10.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:27:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:27:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:27:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:27:10.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:27:10.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:27:10.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:27:10.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:27:10.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:27:10.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:27:10.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:27:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:27:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:10.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:27:11.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 5 2026-02-20T22:27:11.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:27:11.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:27:11.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:11.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:27:11.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:27:11.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:11.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:27:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:27:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:27:11.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:27:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:27:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:27:11.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:27:11.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:11.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:27:11.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:27:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:27:11.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:27:11.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=5 2026-02-20T22:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:27:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:27:11.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:27:11.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:27:11.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:27:11.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:27:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:27:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:27:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:27:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:27:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 5 = '' 2026-02-20T22:27:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:27:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:27:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:27:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:27:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:27:12.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:27:12.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:27:12.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:27:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:27:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 5 = '' 2026-02-20T22:27:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:27:12.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:27:13.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:27:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:27:13.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:27:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:27:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:27:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:27:13.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 5 = '' 2026-02-20T22:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:27:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:27:14.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:27:14.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=5 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 5 = 5 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:27:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:27:15.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:27:15.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:27:15.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap1 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:27:15.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:27:15.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:27:15.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:27:16.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:27:16.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:27:17.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:27:17.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:27:17.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:27:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:17.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap1-clean 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap1-clean 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1-clean 2026-02-20T22:27:17.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:27:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:27:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:27:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:17.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:27:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=12 2026-02-20T22:27:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:27:17.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:27:17.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 25253888 bytes 2026-02-20T22:27:17.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 27069440 bytes 2026-02-20T22:27:17.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 65723392 bytes 2026-02-20T22:27:17.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 102288384 bytes 2026-02-20T22:27:17.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 167782400 bytes 2026-02-20T22:27:17.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 212383744 bytes 2026-02-20T22:27:17.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 213015552 bytes 2026-02-20T22:27:17.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 225716224 bytes 2026-02-20T22:27:18.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 296089600 bytes 2026-02-20T22:27:18.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 361097216 bytes 2026-02-20T22:27:18.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 380558336 bytes 2026-02-20T22:27:18.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 437479424 bytes 2026-02-20T22:27:18.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 465567744 bytes 2026-02-20T22:27:18.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 483840000 bytes 2026-02-20T22:27:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 534389760 bytes 2026-02-20T22:27:18.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 650896384 bytes 2026-02-20T22:27:18.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 704741376 bytes 2026-02-20T22:27:18.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 739940352 bytes 2026-02-20T22:27:18.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 776462336 bytes 2026-02-20T22:27:18.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 811077632 bytes 2026-02-20T22:27:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 847799296 bytes 2026-02-20T22:27:18.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 857023488 bytes 2026-02-20T22:27:18.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 905344000 bytes 2026-02-20T22:27:18.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 938590208 bytes 2026-02-20T22:27:18.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 957820928 bytes 2026-02-20T22:27:18.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 999257088 bytes 2026-02-20T22:27:18.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1069876224 bytes 2026-02-20T22:27:18.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1072914432 bytes 2026-02-20T22:27:18.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1085260800 bytes 2026-02-20T22:27:18.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1115108352 bytes 2026-02-20T22:27:18.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1163310080 bytes 2026-02-20T22:27:18.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1233748992 bytes 2026-02-20T22:27:19.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1285350400 bytes 2026-02-20T22:27:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1285983232 bytes 2026-02-20T22:27:19.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1330779136 bytes 2026-02-20T22:27:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1387125760 bytes 2026-02-20T22:27:19.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1446650880 bytes 2026-02-20T22:27:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1447255040 bytes 2026-02-20T22:27:19.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1484239872 bytes 2026-02-20T22:27:19.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1532568576 bytes 2026-02-20T22:27:19.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1541198848 bytes 2026-02-20T22:27:19.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1585222656 bytes 2026-02-20T22:27:19.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1636833280 bytes 2026-02-20T22:27:19.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1658800128 bytes 2026-02-20T22:27:19.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1724306432 bytes 2026-02-20T22:27:19.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1763897344 bytes 2026-02-20T22:27:19.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1788236800 bytes 2026-02-20T22:27:19.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1827831808 bytes 2026-02-20T22:27:19.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1848285184 bytes 2026-02-20T22:27:19.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1883555840 bytes 2026-02-20T22:27:19.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1892606976 bytes 2026-02-20T22:27:19.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1926006784 bytes 2026-02-20T22:27:19.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1965324288 bytes 2026-02-20T22:27:19.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2021365760 bytes 2026-02-20T22:27:19.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2021981184 bytes 2026-02-20T22:27:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2033013760 bytes 2026-02-20T22:27:19.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2107826176 bytes 2026-02-20T22:27:19.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2122031104 bytes 2026-02-20T22:27:20.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2141816832 bytes 2026-02-20T22:27:20.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2178252800 bytes 2026-02-20T22:27:20.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2179181568 bytes 2026-02-20T22:27:20.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2201925632 bytes 2026-02-20T22:27:20.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2253892608 bytes 2026-02-20T22:27:20.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2284590080 bytes 2026-02-20T22:27:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2348951552 bytes 2026-02-20T22:27:20.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2396708864 bytes 2026-02-20T22:27:20.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2397323264 bytes 2026-02-20T22:27:20.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2414373888 bytes 2026-02-20T22:27:20.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2423170048 bytes 2026-02-20T22:27:20.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2445530112 bytes 2026-02-20T22:27:20.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2466403328 bytes 2026-02-20T22:27:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2531615744 bytes 2026-02-20T22:27:20.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2563305472 bytes 2026-02-20T22:27:20.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2661833728 bytes 2026-02-20T22:27:20.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2679800832 bytes 2026-02-20T22:27:20.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2699496448 bytes 2026-02-20T22:27:20.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2764232704 bytes 2026-02-20T22:27:20.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2805711872 bytes 2026-02-20T22:27:20.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2886302720 bytes 2026-02-20T22:27:20.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 2895557632 bytes 2026-02-20T22:27:20.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 2947877888 bytes 2026-02-20T22:27:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 2974385152 bytes 2026-02-20T22:27:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3016367104 bytes 2026-02-20T22:27:20.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3016973312 bytes 2026-02-20T22:27:20.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3028454400 bytes 2026-02-20T22:27:21.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3050886144 bytes 2026-02-20T22:27:21.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3082646528 bytes 2026-02-20T22:27:21.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3139738624 bytes 2026-02-20T22:27:21.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3202686976 bytes 2026-02-20T22:27:21.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3203309568 bytes 2026-02-20T22:27:21.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3240331264 bytes 2026-02-20T22:27:21.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3260258304 bytes 2026-02-20T22:27:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3264376832 bytes 2026-02-20T22:27:21.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3284553728 bytes 2026-02-20T22:27:21.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3308481536 bytes 2026-02-20T22:27:21.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3379301376 bytes 2026-02-20T22:27:21.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3447074816 bytes 2026-02-20T22:27:21.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3480198144 bytes 2026-02-20T22:27:21.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3513993216 bytes 2026-02-20T22:27:21.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3532304384 bytes 2026-02-20T22:27:21.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3561590784 bytes 2026-02-20T22:27:21.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 3600929792 bytes 2026-02-20T22:27:21.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 3617740800 bytes 2026-02-20T22:27:21.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 3643094016 bytes 2026-02-20T22:27:21.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 3689428992 bytes 2026-02-20T22:27:21.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 3709911040 bytes 2026-02-20T22:27:21.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 3791481856 bytes 2026-02-20T22:27:21.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 3822089216 bytes 2026-02-20T22:27:21.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 3893166080 bytes 2026-02-20T22:27:21.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 3906969600 bytes 2026-02-20T22:27:21.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 3944131584 bytes 2026-02-20T22:27:22.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4009997312 bytes 2026-02-20T22:27:22.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4072171520 bytes 2026-02-20T22:27:22.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4116403200 bytes 2026-02-20T22:27:22.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4118578176 bytes 2026-02-20T22:27:22.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4169176064 bytes 2026-02-20T22:27:22.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4237365248 bytes 2026-02-20T22:27:22.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4326952960 bytes 2026-02-20T22:27:22.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4415938560 bytes 2026-02-20T22:27:22.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4464906240 bytes 2026-02-20T22:27:22.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4493924352 bytes 2026-02-20T22:27:22.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4535280640 bytes 2026-02-20T22:27:22.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4559506432 bytes 2026-02-20T22:27:22.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4560439296 bytes 2026-02-20T22:27:22.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4575167488 bytes 2026-02-20T22:27:22.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4591818752 bytes 2026-02-20T22:27:22.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 4672811008 bytes 2026-02-20T22:27:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 4711622656 bytes 2026-02-20T22:27:22.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 4735825920 bytes 2026-02-20T22:27:22.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 4770273280 bytes 2026-02-20T22:27:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 4862269440 bytes 2026-02-20T22:27:22.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 4912350208 bytes 2026-02-20T22:27:22.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 4973487104 bytes 2026-02-20T22:27:22.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 4986504192 bytes 2026-02-20T22:27:23.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5027673088 bytes 2026-02-20T22:27:23.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5078414336 bytes 2026-02-20T22:27:23.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5132727296 bytes 2026-02-20T22:27:23.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5150932992 bytes 2026-02-20T22:27:23.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5200016384 bytes 2026-02-20T22:27:23.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5222131712 bytes 2026-02-20T22:27:23.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5238301696 bytes 2026-02-20T22:27:23.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5292179456 bytes 2026-02-20T22:27:23.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5321294848 bytes 2026-02-20T22:27:23.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5421822976 bytes 2026-02-20T22:27:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5485805568 bytes 2026-02-20T22:27:23.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5496323072 bytes 2026-02-20T22:27:23.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5511566336 bytes 2026-02-20T22:27:23.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5522499584 bytes 2026-02-20T22:27:23.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 5558651904 bytes 2026-02-20T22:27:23.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 5634661376 bytes 2026-02-20T22:27:23.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 5651142656 bytes 2026-02-20T22:27:23.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 5676856320 bytes 2026-02-20T22:27:23.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 5742065664 bytes 2026-02-20T22:27:23.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 5742701568 bytes 2026-02-20T22:27:23.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 5795137536 bytes 2026-02-20T22:27:23.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 5843892224 bytes 2026-02-20T22:27:23.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 5902280704 bytes 2026-02-20T22:27:24.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 5972789248 bytes 2026-02-20T22:27:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6020842496 bytes 2026-02-20T22:27:24.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6056776704 bytes 2026-02-20T22:27:24.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6091387904 bytes 2026-02-20T22:27:24.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6103240704 bytes 2026-02-20T22:27:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6135072768 bytes 2026-02-20T22:27:24.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6225333248 bytes 2026-02-20T22:27:24.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6298389504 bytes 2026-02-20T22:27:24.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6316001280 bytes 2026-02-20T22:27:24.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6380764160 bytes 2026-02-20T22:27:24.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6518278144 bytes 2026-02-20T22:27:24.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6564661248 bytes 2026-02-20T22:27:24.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 6581087232 bytes 2026-02-20T22:27:24.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 6625267712 bytes 2026-02-20T22:27:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 6680597504 bytes 2026-02-20T22:27:24.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 6714067968 bytes 2026-02-20T22:27:24.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 6760792064 bytes 2026-02-20T22:27:24.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 6809987072 bytes 2026-02-20T22:27:24.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 6842151936 bytes 2026-02-20T22:27:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 6874779648 bytes 2026-02-20T22:27:24.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 6889715712 bytes 2026-02-20T22:27:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 6934218752 bytes 2026-02-20T22:27:25.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 6969908224 bytes 2026-02-20T22:27:25.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 6970526720 bytes 2026-02-20T22:27:25.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7080140800 bytes 2026-02-20T22:27:25.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7144245248 bytes 2026-02-20T22:27:25.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7210714112 bytes 2026-02-20T22:27:25.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7283407872 bytes 2026-02-20T22:27:25.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7313746944 bytes 2026-02-20T22:27:25.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7355877376 bytes 2026-02-20T22:27:25.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6085930183499204) > weight (0.25) 2026-02-20T22:27:25.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8049902970780752) > weight (0.25) 2026-02-20T22:27:25.438 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9220087531314782) > weight (0.25) 2026-02-20T22:27:25.438 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2026-02-20T22:27:25.439 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2026-02-20T22:27:25.439 INFO:tasks.rbd_mirror_thrash:waiting for 2.6309801310673766 secs before reviving daemons 2026-02-20T22:27:25.440 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:27:25.432+0000 7f4eaf7d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 19657) UID: 1000 2026-02-20T22:27:25.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7432759296 bytes 2026-02-20T22:27:25.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7536816128 bytes 2026-02-20T22:27:25.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7585798144 bytes 2026-02-20T22:27:25.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7683814400 bytes 2026-02-20T22:27:25.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7704694784 bytes 2026-02-20T22:27:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7775403008 bytes 2026-02-20T22:27:25.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7845041152 bytes 2026-02-20T22:27:25.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 7847881728 bytes 2026-02-20T22:27:25.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 7872553984 bytes 2026-02-20T22:27:25.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 7909204992 bytes 2026-02-20T22:27:25.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 7966623744 bytes 2026-02-20T22:27:25.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 7985682432 bytes 2026-02-20T22:27:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8021199872 bytes 2026-02-20T22:27:26.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8036405248 bytes 2026-02-20T22:27:26.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8104967168 bytes 2026-02-20T22:27:26.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8172420096 bytes 2026-02-20T22:27:26.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8211846144 bytes 2026-02-20T22:27:26.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8235432960 bytes 2026-02-20T22:27:26.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8251467776 bytes 2026-02-20T22:27:26.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8318203904 bytes 2026-02-20T22:27:26.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8369947648 bytes 2026-02-20T22:27:26.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8386908160 bytes 2026-02-20T22:27:26.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8454534144 bytes 2026-02-20T22:27:26.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8469520384 bytes 2026-02-20T22:27:26.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 8477778944 bytes 2026-02-20T22:27:27.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 8539983872 bytes 2026-02-20T22:27:27.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 8580722688 bytes 2026-02-20T22:27:27.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 8650447872 bytes 2026-02-20T22:27:27.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 8695889920 bytes 2026-02-20T22:27:27.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 8718999552 bytes 2026-02-20T22:27:27.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 8777561088 bytes 2026-02-20T22:27:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 8863437824 bytes 2026-02-20T22:27:27.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 8879431680 bytes 2026-02-20T22:27:27.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 8895766528 bytes 2026-02-20T22:27:27.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 8925584384 bytes 2026-02-20T22:27:27.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 8931989504 bytes 2026-02-20T22:27:27.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 8980097024 bytes 2026-02-20T22:27:27.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9044195328 bytes 2026-02-20T22:27:27.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9086514176 bytes 2026-02-20T22:27:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 9130106880 bytes 2026-02-20T22:27:27.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 9157358592 bytes 2026-02-20T22:27:27.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 9240531968 bytes 2026-02-20T22:27:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 9280000000 bytes 2026-02-20T22:27:27.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 9331257344 bytes 2026-02-20T22:27:28.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23200 IOs, 9364639744 bytes 2026-02-20T22:27:28.071 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2026-02-20T22:27:28.071 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:27:28.071 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2026-02-20T22:27:28.072 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-20T22:27:28.072 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-20T22:27:28.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-20T22:27:28.074 INFO:tasks.rbd_mirror_thrash:waiting for 70.00579414700894 secs before thrashing 2026-02-20T22:27:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23300 IOs, 9392684032 bytes 2026-02-20T22:27:28.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23400 IOs, 9492035584 bytes 2026-02-20T22:27:28.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23500 IOs, 9509388288 bytes 2026-02-20T22:27:28.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23600 IOs, 9522136064 bytes 2026-02-20T22:27:28.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23700 IOs, 9563097088 bytes 2026-02-20T22:27:28.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23800 IOs, 9610855424 bytes 2026-02-20T22:27:28.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23900 IOs, 9639997440 bytes 2026-02-20T22:27:28.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24000 IOs, 9653633024 bytes 2026-02-20T22:27:28.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24100 IOs, 9655921664 bytes 2026-02-20T22:27:28.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24200 IOs, 9721078784 bytes 2026-02-20T22:27:28.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24300 IOs, 9745474560 bytes 2026-02-20T22:27:28.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24400 IOs, 9773514752 bytes 2026-02-20T22:27:28.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24500 IOs, 9852178432 bytes 2026-02-20T22:27:28.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24600 IOs, 9941984256 bytes 2026-02-20T22:27:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24700 IOs, 9977208832 bytes 2026-02-20T22:27:28.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24800 IOs, 9977818112 bytes 2026-02-20T22:27:28.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24900 IOs, 10011526144 bytes 2026-02-20T22:27:28.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25000 IOs, 10032447488 bytes 2026-02-20T22:27:29.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25100 IOs, 10114365440 bytes 2026-02-20T22:27:29.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25200 IOs, 10164819968 bytes 2026-02-20T22:27:29.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25300 IOs, 10182922240 bytes 2026-02-20T22:27:29.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25400 IOs, 10242583552 bytes 2026-02-20T22:27:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25500 IOs, 10254743552 bytes 2026-02-20T22:27:29.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25600 IOs, 10293612544 bytes 2026-02-20T22:27:29.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25700 IOs, 10353034240 bytes 2026-02-20T22:27:29.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25800 IOs, 10409788416 bytes 2026-02-20T22:27:29.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:25900 IOs, 10410413056 bytes 2026-02-20T22:27:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:26000 IOs, 10429678592 bytes 2026-02-20T22:27:29.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:26100 IOs, 10451401728 bytes 2026-02-20T22:27:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:27:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:27:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:27:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:27:29.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:27:30.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:27:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":17888597.33,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"} 2026-02-20T22:27:30.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:27:27 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:27:26' 2026-02-20T22:27:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:27:30.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:27:30.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":17888597.33,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"} 2026-02-20T22:27:30.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:27:30.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:27:30.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:27:30.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:27:30.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:27:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:27:27' 2026-02-20T22:27:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:27:27' 2026-02-20T22:27:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:30.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:30.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:27:30.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:27:27' 2026-02-20T22:27:30.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:27:30.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:27:30.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:27:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":17888597.33,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:27:27cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:27:26' 2026-02-20T22:27:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:27:30.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:27:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:27:27' 2026-02-20T22:27:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:27:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:27:30.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:27:30.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:27:30.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap2 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap2 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:30.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2 2026-02-20T22:27:30.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:27:59.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:27:59.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:59.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:27:59.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:28:00.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:28:00.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:00.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:28:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:28:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:28:00.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:28:00.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:28:00.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:28:00.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:28:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:28:00.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:28:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:00.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:00.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:28:00.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:28:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:28:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:28:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:28:00.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:00.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:28:02.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 8 2026-02-20T22:28:02.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:02.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:28:02.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:28:02.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:28:02.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:02.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:28:02.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:28:02.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:28:02.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:28:02.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:02.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:28:02.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=8 2026-02-20T22:28:02.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:02.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:02.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:28:02.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:02.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:02.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:02.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:28:02.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 8 = '' 2026-02-20T22:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:02.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:03.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:03.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:28:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:28:03.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:28:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:28:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 8 = '' 2026-02-20T22:28:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:03.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:28:04.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:04.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:28:04.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:28:04.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:28:04.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:28:04.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 8 = '' 2026-02-20T22:28:04.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:04.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:28:05.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:05.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=8 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 8 = 8 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2 2026-02-20T22:28:05.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:28:05.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:28:05.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:28:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 4 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:28:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:28:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:28:06.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap1-clean ']' 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap1-clean 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1-clean 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:06.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:06.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:07.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:07.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:07.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:07.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:08.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:08.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap2 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:08.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:08.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:08.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:09.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:09.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap2-clean 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap2-clean 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:09.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2-clean 2026-02-20T22:28:09.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:09.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:28:09.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:28:09.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:28:09.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:28:09.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 22646784 bytes 2026-02-20T22:28:09.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 53424128 bytes 2026-02-20T22:28:09.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 87415808 bytes 2026-02-20T22:28:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 131883008 bytes 2026-02-20T22:28:09.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 146179072 bytes 2026-02-20T22:28:09.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 152843264 bytes 2026-02-20T22:28:10.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 185811968 bytes 2026-02-20T22:28:10.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 224658432 bytes 2026-02-20T22:28:10.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 279522304 bytes 2026-02-20T22:28:10.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 344907776 bytes 2026-02-20T22:28:10.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 402118656 bytes 2026-02-20T22:28:10.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 453964800 bytes 2026-02-20T22:28:10.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 467890176 bytes 2026-02-20T22:28:10.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 493122560 bytes 2026-02-20T22:28:10.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 517214208 bytes 2026-02-20T22:28:10.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 579284992 bytes 2026-02-20T22:28:10.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 594604032 bytes 2026-02-20T22:28:10.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 632349696 bytes 2026-02-20T22:28:10.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 643151872 bytes 2026-02-20T22:28:10.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 714948608 bytes 2026-02-20T22:28:10.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 754684928 bytes 2026-02-20T22:28:10.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 755277824 bytes 2026-02-20T22:28:10.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 799181824 bytes 2026-02-20T22:28:10.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 840729600 bytes 2026-02-20T22:28:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 903243776 bytes 2026-02-20T22:28:10.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 945823744 bytes 2026-02-20T22:28:10.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1015027712 bytes 2026-02-20T22:28:10.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1042957312 bytes 2026-02-20T22:28:10.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1088161792 bytes 2026-02-20T22:28:11.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1107660800 bytes 2026-02-20T22:28:11.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1145220096 bytes 2026-02-20T22:28:11.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1194452992 bytes 2026-02-20T22:28:11.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1235993600 bytes 2026-02-20T22:28:11.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1318086656 bytes 2026-02-20T22:28:11.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1352429568 bytes 2026-02-20T22:28:11.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1364864000 bytes 2026-02-20T22:28:11.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1403565056 bytes 2026-02-20T22:28:11.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1417560064 bytes 2026-02-20T22:28:11.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1471713280 bytes 2026-02-20T22:28:11.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1502610432 bytes 2026-02-20T22:28:11.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1503242240 bytes 2026-02-20T22:28:11.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1546985472 bytes 2026-02-20T22:28:11.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1614499840 bytes 2026-02-20T22:28:11.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1674783744 bytes 2026-02-20T22:28:11.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1693499392 bytes 2026-02-20T22:28:11.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1758013440 bytes 2026-02-20T22:28:11.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1859268608 bytes 2026-02-20T22:28:11.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1931810816 bytes 2026-02-20T22:28:11.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1989661696 bytes 2026-02-20T22:28:11.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2006699008 bytes 2026-02-20T22:28:11.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2040457216 bytes 2026-02-20T22:28:12.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2077880320 bytes 2026-02-20T22:28:12.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2124265472 bytes 2026-02-20T22:28:12.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2168412160 bytes 2026-02-20T22:28:12.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2216653824 bytes 2026-02-20T22:28:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2250754048 bytes 2026-02-20T22:28:12.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2369435648 bytes 2026-02-20T22:28:12.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2416260096 bytes 2026-02-20T22:28:12.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2474319872 bytes 2026-02-20T22:28:12.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2507056128 bytes 2026-02-20T22:28:12.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2572178432 bytes 2026-02-20T22:28:12.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2629853184 bytes 2026-02-20T22:28:12.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2689210368 bytes 2026-02-20T22:28:12.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2727878656 bytes 2026-02-20T22:28:12.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2769707008 bytes 2026-02-20T22:28:12.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2801944576 bytes 2026-02-20T22:28:12.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2843261952 bytes 2026-02-20T22:28:12.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2879427584 bytes 2026-02-20T22:28:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2946232320 bytes 2026-02-20T22:28:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2967511040 bytes 2026-02-20T22:28:12.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3030025216 bytes 2026-02-20T22:28:13.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3044528128 bytes 2026-02-20T22:28:13.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3071436800 bytes 2026-02-20T22:28:13.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3122827264 bytes 2026-02-20T22:28:13.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3133622272 bytes 2026-02-20T22:28:13.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3240015872 bytes 2026-02-20T22:28:13.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3254132736 bytes 2026-02-20T22:28:13.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3309786112 bytes 2026-02-20T22:28:13.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3327523840 bytes 2026-02-20T22:28:13.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3360088064 bytes 2026-02-20T22:28:13.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3385344000 bytes 2026-02-20T22:28:13.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3459328000 bytes 2026-02-20T22:28:13.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3530190848 bytes 2026-02-20T22:28:13.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3612721152 bytes 2026-02-20T22:28:13.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3681916928 bytes 2026-02-20T22:28:13.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3734028288 bytes 2026-02-20T22:28:13.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3779956736 bytes 2026-02-20T22:28:13.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3917086720 bytes 2026-02-20T22:28:13.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4003528704 bytes 2026-02-20T22:28:13.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4074902528 bytes 2026-02-20T22:28:14.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4121582592 bytes 2026-02-20T22:28:14.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4171223040 bytes 2026-02-20T22:28:14.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4227314688 bytes 2026-02-20T22:28:14.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4249865216 bytes 2026-02-20T22:28:14.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4299265024 bytes 2026-02-20T22:28:14.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4299879424 bytes 2026-02-20T22:28:14.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4325098496 bytes 2026-02-20T22:28:14.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4383495168 bytes 2026-02-20T22:28:14.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4406219776 bytes 2026-02-20T22:28:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4456239104 bytes 2026-02-20T22:28:14.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4524366848 bytes 2026-02-20T22:28:14.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4569410560 bytes 2026-02-20T22:28:14.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4623220736 bytes 2026-02-20T22:28:14.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4664720384 bytes 2026-02-20T22:28:14.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4693730304 bytes 2026-02-20T22:28:14.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4736404480 bytes 2026-02-20T22:28:14.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4795060224 bytes 2026-02-20T22:28:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4820391936 bytes 2026-02-20T22:28:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4831651840 bytes 2026-02-20T22:28:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4897203200 bytes 2026-02-20T22:28:14.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4915811328 bytes 2026-02-20T22:28:14.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4962266112 bytes 2026-02-20T22:28:15.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5010990080 bytes 2026-02-20T22:28:15.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5086110720 bytes 2026-02-20T22:28:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5152772096 bytes 2026-02-20T22:28:15.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5194560512 bytes 2026-02-20T22:28:15.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5251765248 bytes 2026-02-20T22:28:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5319603200 bytes 2026-02-20T22:28:15.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5368765440 bytes 2026-02-20T22:28:15.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5396939776 bytes 2026-02-20T22:28:15.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5407403008 bytes 2026-02-20T22:28:15.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5491096576 bytes 2026-02-20T22:28:15.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5519949824 bytes 2026-02-20T22:28:15.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5543021568 bytes 2026-02-20T22:28:15.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5604368384 bytes 2026-02-20T22:28:15.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5678536704 bytes 2026-02-20T22:28:15.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5701303296 bytes 2026-02-20T22:28:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5759807488 bytes 2026-02-20T22:28:15.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5760438272 bytes 2026-02-20T22:28:15.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5793332224 bytes 2026-02-20T22:28:15.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5822466048 bytes 2026-02-20T22:28:15.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5846877184 bytes 2026-02-20T22:28:16.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5911185408 bytes 2026-02-20T22:28:16.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5935430656 bytes 2026-02-20T22:28:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5967038464 bytes 2026-02-20T22:28:16.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6000351232 bytes 2026-02-20T22:28:16.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6044912640 bytes 2026-02-20T22:28:16.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6091654144 bytes 2026-02-20T22:28:16.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6104871936 bytes 2026-02-20T22:28:16.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6156313600 bytes 2026-02-20T22:28:16.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6205239296 bytes 2026-02-20T22:28:16.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6229343232 bytes 2026-02-20T22:28:16.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6290579456 bytes 2026-02-20T22:28:16.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6311035904 bytes 2026-02-20T22:28:16.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6381443072 bytes 2026-02-20T22:28:16.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6462875648 bytes 2026-02-20T22:28:16.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6585844736 bytes 2026-02-20T22:28:16.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6619214848 bytes 2026-02-20T22:28:16.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6667747328 bytes 2026-02-20T22:28:16.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6715370496 bytes 2026-02-20T22:28:16.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6735880192 bytes 2026-02-20T22:28:16.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6750774272 bytes 2026-02-20T22:28:16.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6778892288 bytes 2026-02-20T22:28:16.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6779499520 bytes 2026-02-20T22:28:17.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6826280960 bytes 2026-02-20T22:28:17.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6855231488 bytes 2026-02-20T22:28:17.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6934584320 bytes 2026-02-20T22:28:17.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6998473728 bytes 2026-02-20T22:28:17.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7007904768 bytes 2026-02-20T22:28:17.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7040887808 bytes 2026-02-20T22:28:17.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7060773888 bytes 2026-02-20T22:28:17.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7109361664 bytes 2026-02-20T22:28:17.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7135731712 bytes 2026-02-20T22:28:17.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7179549696 bytes 2026-02-20T22:28:17.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7217825792 bytes 2026-02-20T22:28:17.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7241317376 bytes 2026-02-20T22:28:17.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7284619264 bytes 2026-02-20T22:28:17.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7346424832 bytes 2026-02-20T22:28:17.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7426620416 bytes 2026-02-20T22:28:17.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7451678720 bytes 2026-02-20T22:28:17.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7462564864 bytes 2026-02-20T22:28:17.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7486771200 bytes 2026-02-20T22:28:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7520397312 bytes 2026-02-20T22:28:17.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7556391936 bytes 2026-02-20T22:28:17.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7638169600 bytes 2026-02-20T22:28:18.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7659243520 bytes 2026-02-20T22:28:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7694986240 bytes 2026-02-20T22:28:18.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7749373952 bytes 2026-02-20T22:28:18.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7768148992 bytes 2026-02-20T22:28:18.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7853127680 bytes 2026-02-20T22:28:18.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7883055104 bytes 2026-02-20T22:28:18.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7975912448 bytes 2026-02-20T22:28:18.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8018867200 bytes 2026-02-20T22:28:18.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8057592832 bytes 2026-02-20T22:28:18.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8083192832 bytes 2026-02-20T22:28:18.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8102066176 bytes 2026-02-20T22:28:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8153485312 bytes 2026-02-20T22:28:18.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8177573888 bytes 2026-02-20T22:28:18.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8206703616 bytes 2026-02-20T22:28:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8217585664 bytes 2026-02-20T22:28:18.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8242534400 bytes 2026-02-20T22:28:18.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8263084032 bytes 2026-02-20T22:28:18.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8338725888 bytes 2026-02-20T22:28:18.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8368432128 bytes 2026-02-20T22:28:18.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8429049856 bytes 2026-02-20T22:28:18.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8473242624 bytes 2026-02-20T22:28:18.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8487753728 bytes 2026-02-20T22:28:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8540157952 bytes 2026-02-20T22:28:19.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8549320704 bytes 2026-02-20T22:28:19.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8593552384 bytes 2026-02-20T22:28:19.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8636872704 bytes 2026-02-20T22:28:19.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8710808576 bytes 2026-02-20T22:28:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8711512064 bytes 2026-02-20T22:28:19.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8729018368 bytes 2026-02-20T22:28:19.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8733258752 bytes 2026-02-20T22:28:19.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8804700160 bytes 2026-02-20T22:28:19.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8850295808 bytes 2026-02-20T22:28:19.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8883691520 bytes 2026-02-20T22:28:19.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8929554432 bytes 2026-02-20T22:28:19.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8932800512 bytes 2026-02-20T22:28:19.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8980666368 bytes 2026-02-20T22:28:19.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9042469888 bytes 2026-02-20T22:28:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:28:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:28:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:28:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:28:19.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:28:20.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:28:20.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"} 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:27:57 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:27:56' 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:28:20.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:28:20.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:28:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"} 2026-02-20T22:28:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:28:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:28:20.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:28:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:28:20.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:28:20.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:28:20.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:28:20.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:27:57' 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:27:57' 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:27:57' 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:28:20.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:28:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":268328960.0,"last_snapshot_bytes":536657920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626431,"remote_snapshot_timestamp":1771626431,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:27:57cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:27:56' 2026-02-20T22:28:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:28:20.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:28:20.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:27:57' 2026-02-20T22:28:20.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:28:20.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:28:20.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:28:20.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:28:20.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap3 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap3 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:20.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:20.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3 2026-02-20T22:28:20.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:28:38.080 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5840145783767529) > weight (0.25) 2026-02-20T22:28:38.080 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-20T22:28:38.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-20T22:28:38.081 INFO:tasks.rbd_mirror_thrash:waiting for 9.104829063834382 secs before reviving daemons 2026-02-20T22:28:38.081 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:28:38.078+0000 7f5bdc4ff640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 19652) UID: 1000 2026-02-20T22:28:47.186 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-20T22:28:47.186 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:28:47.187 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-20T22:28:47.187 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-20T22:28:47.187 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-20T22:28:47.189 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-20T22:28:47.190 INFO:tasks.rbd_mirror_thrash:waiting for 116.62559759263922 secs before thrashing 2026-02-20T22:28:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:28:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:50.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:28:50.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:28:51.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:28:51.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:28:51.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:28:51.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:28:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:28:51.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:28:51.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:28:51.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:28:51.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:28:51.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:28:51.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:51.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:51.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:28:51.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:28:51.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:28:51.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:28:51.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:28:51.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:28:51.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:28:51.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:28:51.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:51.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:51.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:28:52.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7979798346805758) > weight (0.25) 2026-02-20T22:28:52.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6499356007574163) > weight (0.25) 2026-02-20T22:28:52.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8835399632297329) > weight (0.25) 2026-02-20T22:28:52.565 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.624330176026982) > weight (0.25) 2026-02-20T22:28:52.565 INFO:tasks.rbd_mirror_thrash:waiting for 99.46424244130338 secs before thrashing 2026-02-20T22:28:53.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 11 2026-02-20T22:28:53.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:28:53.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:28:53.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:28:53.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:53.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:28:53.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:28:53.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:28:53.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:53.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:53.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:28:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:28:53.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=11 2026-02-20T22:28:53.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:53.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:53.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:28:53.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:53.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:53.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:53.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:53.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:28:53.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:28:53.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:28:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:28:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 11 = '' 2026-02-20T22:28:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:54.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:54.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:28:54.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:28:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:28:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:28:54.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:28:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 11 = '' 2026-02-20T22:28:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:28:54.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:28:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:55.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:28:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:28:55.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:28:55.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:28:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=11 2026-02-20T22:28:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 11 = 11 2026-02-20T22:28:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:28:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:28:56.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2026-02-20T22:28:56.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:28:56.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:28:56.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 6 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:28:07 2026 2026-02-20T22:28:56.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:28:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:28:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:28:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:28:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:28:56.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:28:56.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap2-clean ']' 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap2-clean 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2-clean 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:56.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:56.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:56.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:56.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:57.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:57.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:57.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap3 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:28:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:57.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:57.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:57.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:28:58.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:58.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:58.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:28:58.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:58.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:28:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:28:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap3-clean 2026-02-20T22:28:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap3-clean 2026-02-20T22:28:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3-clean 2026-02-20T22:28:58.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:28:59.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:28:59.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:28:59.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=11 2026-02-20T22:28:59.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:28:59.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:28:59.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 70631424 bytes 2026-02-20T22:28:59.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 82344960 bytes 2026-02-20T22:28:59.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 97263616 bytes 2026-02-20T22:28:59.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 109210624 bytes 2026-02-20T22:29:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 148322304 bytes 2026-02-20T22:29:00.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 168878080 bytes 2026-02-20T22:29:00.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 210969600 bytes 2026-02-20T22:29:00.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 257506304 bytes 2026-02-20T22:29:00.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 306226176 bytes 2026-02-20T22:29:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 332388352 bytes 2026-02-20T22:29:00.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 351433728 bytes 2026-02-20T22:29:00.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 394261504 bytes 2026-02-20T22:29:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 545713152 bytes 2026-02-20T22:29:00.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 604866560 bytes 2026-02-20T22:29:00.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 659011584 bytes 2026-02-20T22:29:00.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 697069568 bytes 2026-02-20T22:29:00.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 726426624 bytes 2026-02-20T22:29:00.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 763091968 bytes 2026-02-20T22:29:00.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 790302720 bytes 2026-02-20T22:29:00.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 880793600 bytes 2026-02-20T22:29:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 937069568 bytes 2026-02-20T22:29:00.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 943533056 bytes 2026-02-20T22:29:00.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 974409728 bytes 2026-02-20T22:29:00.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1028588544 bytes 2026-02-20T22:29:00.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1046441984 bytes 2026-02-20T22:29:00.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1068953600 bytes 2026-02-20T22:29:00.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1072510976 bytes 2026-02-20T22:29:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1101888512 bytes 2026-02-20T22:29:01.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1164552192 bytes 2026-02-20T22:29:01.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1225370624 bytes 2026-02-20T22:29:01.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1226006528 bytes 2026-02-20T22:29:01.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1247000576 bytes 2026-02-20T22:29:01.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1300047872 bytes 2026-02-20T22:29:01.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1309121536 bytes 2026-02-20T22:29:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1348971520 bytes 2026-02-20T22:29:01.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1409700864 bytes 2026-02-20T22:29:01.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1485903872 bytes 2026-02-20T22:29:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1526189056 bytes 2026-02-20T22:29:01.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1554502656 bytes 2026-02-20T22:29:01.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1608377344 bytes 2026-02-20T22:29:01.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1625771008 bytes 2026-02-20T22:29:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1666796544 bytes 2026-02-20T22:29:01.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1702287360 bytes 2026-02-20T22:29:01.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1789032448 bytes 2026-02-20T22:29:01.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1863837696 bytes 2026-02-20T22:29:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1889266688 bytes 2026-02-20T22:29:01.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1952547840 bytes 2026-02-20T22:29:01.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1987364864 bytes 2026-02-20T22:29:02.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2028264448 bytes 2026-02-20T22:29:02.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2052752384 bytes 2026-02-20T22:29:02.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2094766080 bytes 2026-02-20T22:29:02.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2138272768 bytes 2026-02-20T22:29:02.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2211814400 bytes 2026-02-20T22:29:02.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2265162752 bytes 2026-02-20T22:29:02.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2323354624 bytes 2026-02-20T22:29:02.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2390760448 bytes 2026-02-20T22:29:02.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2433704960 bytes 2026-02-20T22:29:02.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2515162112 bytes 2026-02-20T22:29:02.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2533965824 bytes 2026-02-20T22:29:02.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2590449664 bytes 2026-02-20T22:29:02.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2652724224 bytes 2026-02-20T22:29:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2721613824 bytes 2026-02-20T22:29:02.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2740594688 bytes 2026-02-20T22:29:02.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2791814144 bytes 2026-02-20T22:29:02.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2805063680 bytes 2026-02-20T22:29:02.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2859077632 bytes 2026-02-20T22:29:02.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2859698176 bytes 2026-02-20T22:29:02.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2891983872 bytes 2026-02-20T22:29:03.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2996436992 bytes 2026-02-20T22:29:03.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3021026304 bytes 2026-02-20T22:29:03.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3098654720 bytes 2026-02-20T22:29:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3112861696 bytes 2026-02-20T22:29:03.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3175264256 bytes 2026-02-20T22:29:03.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3220846592 bytes 2026-02-20T22:29:03.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3251235840 bytes 2026-02-20T22:29:03.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3317155840 bytes 2026-02-20T22:29:03.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3333023744 bytes 2026-02-20T22:29:03.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3359360000 bytes 2026-02-20T22:29:03.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3374130176 bytes 2026-02-20T22:29:03.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3409148928 bytes 2026-02-20T22:29:03.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3438311424 bytes 2026-02-20T22:29:03.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3459154944 bytes 2026-02-20T22:29:03.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3536226304 bytes 2026-02-20T22:29:03.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3566794752 bytes 2026-02-20T22:29:03.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3646029824 bytes 2026-02-20T22:29:03.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3700187136 bytes 2026-02-20T22:29:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3735220224 bytes 2026-02-20T22:29:03.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3755636736 bytes 2026-02-20T22:29:03.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3823684608 bytes 2026-02-20T22:29:03.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3887324160 bytes 2026-02-20T22:29:04.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3893277696 bytes 2026-02-20T22:29:04.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3952224256 bytes 2026-02-20T22:29:04.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4017961984 bytes 2026-02-20T22:29:04.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4089410560 bytes 2026-02-20T22:29:04.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4161927168 bytes 2026-02-20T22:29:04.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4186927104 bytes 2026-02-20T22:29:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4190848000 bytes 2026-02-20T22:29:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4231018496 bytes 2026-02-20T22:29:04.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4236603392 bytes 2026-02-20T22:29:04.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4237203456 bytes 2026-02-20T22:29:04.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4299410432 bytes 2026-02-20T22:29:04.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4370286592 bytes 2026-02-20T22:29:04.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4472773632 bytes 2026-02-20T22:29:04.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4529487872 bytes 2026-02-20T22:29:04.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4530091008 bytes 2026-02-20T22:29:04.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4538352640 bytes 2026-02-20T22:29:04.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4573546496 bytes 2026-02-20T22:29:04.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4658109440 bytes 2026-02-20T22:29:04.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4696850432 bytes 2026-02-20T22:29:05.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4792042496 bytes 2026-02-20T22:29:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4793554944 bytes 2026-02-20T22:29:05.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4814902272 bytes 2026-02-20T22:29:05.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4831586304 bytes 2026-02-20T22:29:05.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4893641728 bytes 2026-02-20T22:29:05.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5004334080 bytes 2026-02-20T22:29:05.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5118536704 bytes 2026-02-20T22:29:05.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5131367424 bytes 2026-02-20T22:29:05.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5169509376 bytes 2026-02-20T22:29:05.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5312977920 bytes 2026-02-20T22:29:05.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5387373568 bytes 2026-02-20T22:29:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5426951168 bytes 2026-02-20T22:29:05.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5485694976 bytes 2026-02-20T22:29:05.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5547891712 bytes 2026-02-20T22:29:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5599083520 bytes 2026-02-20T22:29:05.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5644999680 bytes 2026-02-20T22:29:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5670809600 bytes 2026-02-20T22:29:05.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5713671168 bytes 2026-02-20T22:29:05.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5765842944 bytes 2026-02-20T22:29:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5819038720 bytes 2026-02-20T22:29:06.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5836325888 bytes 2026-02-20T22:29:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5860427776 bytes 2026-02-20T22:29:06.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5883084800 bytes 2026-02-20T22:29:06.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5963700224 bytes 2026-02-20T22:29:06.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6026596352 bytes 2026-02-20T22:29:06.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6113672192 bytes 2026-02-20T22:29:06.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6140262400 bytes 2026-02-20T22:29:06.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6161718272 bytes 2026-02-20T22:29:06.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6177766400 bytes 2026-02-20T22:29:06.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6226291712 bytes 2026-02-20T22:29:06.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6250646528 bytes 2026-02-20T22:29:06.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6324694016 bytes 2026-02-20T22:29:06.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6360156160 bytes 2026-02-20T22:29:06.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6365015040 bytes 2026-02-20T22:29:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6394979328 bytes 2026-02-20T22:29:06.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6446271488 bytes 2026-02-20T22:29:06.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6446882816 bytes 2026-02-20T22:29:06.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6498326528 bytes 2026-02-20T22:29:06.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6513468416 bytes 2026-02-20T22:29:06.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6555802624 bytes 2026-02-20T22:29:06.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6605368320 bytes 2026-02-20T22:29:07.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6636890112 bytes 2026-02-20T22:29:07.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6662320128 bytes 2026-02-20T22:29:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6707039232 bytes 2026-02-20T22:29:07.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6766394368 bytes 2026-02-20T22:29:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6795166720 bytes 2026-02-20T22:29:07.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6865073152 bytes 2026-02-20T22:29:07.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6877959168 bytes 2026-02-20T22:29:07.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6967177216 bytes 2026-02-20T22:29:07.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7055970304 bytes 2026-02-20T22:29:07.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7107718144 bytes 2026-02-20T22:29:07.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7138635776 bytes 2026-02-20T22:29:07.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7202961408 bytes 2026-02-20T22:29:07.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7279639552 bytes 2026-02-20T22:29:07.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7302713344 bytes 2026-02-20T22:29:07.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7353478144 bytes 2026-02-20T22:29:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7406752768 bytes 2026-02-20T22:29:07.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7449945088 bytes 2026-02-20T22:29:07.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7503160320 bytes 2026-02-20T22:29:07.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7525388288 bytes 2026-02-20T22:29:07.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7567758336 bytes 2026-02-20T22:29:08.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7633555456 bytes 2026-02-20T22:29:08.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7681570816 bytes 2026-02-20T22:29:08.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7696462848 bytes 2026-02-20T22:29:08.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7760975872 bytes 2026-02-20T22:29:08.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7783940096 bytes 2026-02-20T22:29:08.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7817051136 bytes 2026-02-20T22:29:08.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7903541248 bytes 2026-02-20T22:29:08.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7910113280 bytes 2026-02-20T22:29:08.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7938442240 bytes 2026-02-20T22:29:08.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7964860416 bytes 2026-02-20T22:29:08.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8019963904 bytes 2026-02-20T22:29:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8088423424 bytes 2026-02-20T22:29:08.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8089027584 bytes 2026-02-20T22:29:08.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8148299776 bytes 2026-02-20T22:29:08.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8170366976 bytes 2026-02-20T22:29:08.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8181739520 bytes 2026-02-20T22:29:08.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8222163968 bytes 2026-02-20T22:29:08.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8250765312 bytes 2026-02-20T22:29:08.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8342436864 bytes 2026-02-20T22:29:08.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8358754304 bytes 2026-02-20T22:29:09.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8408292352 bytes 2026-02-20T22:29:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8414959616 bytes 2026-02-20T22:29:09.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8426898432 bytes 2026-02-20T22:29:09.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8460386304 bytes 2026-02-20T22:29:09.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8505927680 bytes 2026-02-20T22:29:09.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8522483712 bytes 2026-02-20T22:29:09.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8526747648 bytes 2026-02-20T22:29:09.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8542434304 bytes 2026-02-20T22:29:09.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8614054912 bytes 2026-02-20T22:29:09.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8676804608 bytes 2026-02-20T22:29:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8683853824 bytes 2026-02-20T22:29:09.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8752193536 bytes 2026-02-20T22:29:09.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8788232192 bytes 2026-02-20T22:29:09.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8829003776 bytes 2026-02-20T22:29:09.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8844106752 bytes 2026-02-20T22:29:09.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8856418304 bytes 2026-02-20T22:29:09.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8896537600 bytes 2026-02-20T22:29:09.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8913677312 bytes 2026-02-20T22:29:09.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8978735104 bytes 2026-02-20T22:29:09.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9024744448 bytes 2026-02-20T22:29:10.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9056626688 bytes 2026-02-20T22:29:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9072712704 bytes 2026-02-20T22:29:10.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9112928256 bytes 2026-02-20T22:29:10.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9130904576 bytes 2026-02-20T22:29:10.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9220148224 bytes 2026-02-20T22:29:10.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9302463488 bytes 2026-02-20T22:29:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9366812672 bytes 2026-02-20T22:29:10.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9384493056 bytes 2026-02-20T22:29:10.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9455200256 bytes 2026-02-20T22:29:10.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9504234496 bytes 2026-02-20T22:29:10.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9511503872 bytes 2026-02-20T22:29:10.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9588569088 bytes 2026-02-20T22:29:10.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9655014400 bytes 2026-02-20T22:29:10.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:10.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:29:10.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:29:11.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:29:11.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:29:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":10345949.87,"bytes_per_snapshot":536414720.0,"last_snapshot_bytes":536017920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626532,"remote_snapshot_timestamp":1771626532,"replay_state":"idle"} 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:28:57 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:29:11.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:28:56' 2026-02-20T22:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:29:11.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:29:11.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":10345949.87,"bytes_per_snapshot":536414720.0,"last_snapshot_bytes":536017920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626532,"remote_snapshot_timestamp":1771626532,"replay_state":"idle"} 2026-02-20T22:29:11.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:29:11.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:29:11.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:29:11.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:29:11.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:29:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:29:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:29:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:28:57' 2026-02-20T22:29:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:28:57' 2026-02-20T22:29:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:28:57' 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:29:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:29:11.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":10345949.87,"bytes_per_snapshot":536414720.0,"last_snapshot_bytes":536017920,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626532,"remote_snapshot_timestamp":1771626532,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:28:57cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:28:56' 2026-02-20T22:29:11.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:29:11.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:29:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:28:57' 2026-02-20T22:29:11.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:29:11.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:29:11.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:29:11.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:29:11.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:29:11.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap4 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap4 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4 2026-02-20T22:29:11.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:29:41.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:41.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:29:41.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:29:41.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:29:41.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:29:42.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:29:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:29:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:29:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:29:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:29:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:29:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:29:42.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:29:42.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:29:42.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:29:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:29:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:29:42.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:29:42.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:29:42.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:29:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:29:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:29:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:29:42.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:29:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:29:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:29:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:29:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:29:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 14 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:29:44.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:29:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:29:44.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:29:44.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:29:44.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:29:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:29:44.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:44.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:29:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:29:44.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:29:44.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:29:44.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=14 2026-02-20T22:29:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:29:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:29:44.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:29:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:45.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:29:45.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:29:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:29:45.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:29:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:29:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:29:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:29:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:29:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 14 = '' 2026-02-20T22:29:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:29:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:29:45.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:45.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:29:45.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:29:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:29:45.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:29:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 14 = '' 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:29:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:29:46.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:46.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:46.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:46.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:46.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:29:46.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:29:46.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:29:46.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:29:46.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=14 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 14 = 14 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:29:46.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:29:47.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2026-02-20T22:29:47.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 8 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:29:47.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:28:07 2026 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:29:47.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap3-clean ']' 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap3-clean 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3-clean 2026-02-20T22:29:47.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:29:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:29:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:47.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:29:48.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:29:48.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:48.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:29:48.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:48.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap4 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:48.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:29:49.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:29:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:29:49.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:50.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap4-clean 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap4-clean 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4-clean 2026-02-20T22:29:50.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:29:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:29:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:29:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:50.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:29:50.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:29:50.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:29:50.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:29:50.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 53082112 bytes 2026-02-20T22:29:51.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 135543808 bytes 2026-02-20T22:29:51.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 190116864 bytes 2026-02-20T22:29:51.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 242152448 bytes 2026-02-20T22:29:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 247393280 bytes 2026-02-20T22:29:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 280042496 bytes 2026-02-20T22:29:51.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 306275328 bytes 2026-02-20T22:29:51.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 372293632 bytes 2026-02-20T22:29:51.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 431090688 bytes 2026-02-20T22:29:51.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 477933568 bytes 2026-02-20T22:29:51.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 533650432 bytes 2026-02-20T22:29:51.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 576306176 bytes 2026-02-20T22:29:51.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 635686912 bytes 2026-02-20T22:29:51.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 674593792 bytes 2026-02-20T22:29:51.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 747751424 bytes 2026-02-20T22:29:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 816119808 bytes 2026-02-20T22:29:51.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 844399616 bytes 2026-02-20T22:29:51.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 892277760 bytes 2026-02-20T22:29:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 931291136 bytes 2026-02-20T22:29:51.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 952309760 bytes 2026-02-20T22:29:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 988664832 bytes 2026-02-20T22:29:52.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1025406976 bytes 2026-02-20T22:29:52.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1094811648 bytes 2026-02-20T22:29:52.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1100617728 bytes 2026-02-20T22:29:52.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1143180288 bytes 2026-02-20T22:29:52.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1229765632 bytes 2026-02-20T22:29:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1262148608 bytes 2026-02-20T22:29:52.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1308382208 bytes 2026-02-20T22:29:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1363738624 bytes 2026-02-20T22:29:52.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1414853632 bytes 2026-02-20T22:29:52.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1451302912 bytes 2026-02-20T22:29:52.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1489892352 bytes 2026-02-20T22:29:52.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1555140608 bytes 2026-02-20T22:29:52.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1598772224 bytes 2026-02-20T22:29:52.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1661475840 bytes 2026-02-20T22:29:52.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1742600192 bytes 2026-02-20T22:29:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1821470720 bytes 2026-02-20T22:29:53.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1840411648 bytes 2026-02-20T22:29:53.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1866603520 bytes 2026-02-20T22:29:53.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1932285952 bytes 2026-02-20T22:29:53.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 2060648448 bytes 2026-02-20T22:29:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2172760064 bytes 2026-02-20T22:29:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2215327744 bytes 2026-02-20T22:29:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2286633984 bytes 2026-02-20T22:29:53.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2348843008 bytes 2026-02-20T22:29:53.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2413074432 bytes 2026-02-20T22:29:53.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2465247232 bytes 2026-02-20T22:29:53.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2495704064 bytes 2026-02-20T22:29:53.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2566789120 bytes 2026-02-20T22:29:53.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2601533440 bytes 2026-02-20T22:29:53.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2626202624 bytes 2026-02-20T22:29:53.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2691895296 bytes 2026-02-20T22:29:53.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2775904256 bytes 2026-02-20T22:29:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2819275776 bytes 2026-02-20T22:29:54.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2841398272 bytes 2026-02-20T22:29:54.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2884992000 bytes 2026-02-20T22:29:54.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2996229120 bytes 2026-02-20T22:29:54.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 3012709376 bytes 2026-02-20T22:29:54.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 3100965888 bytes 2026-02-20T22:29:54.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 3136760832 bytes 2026-02-20T22:29:54.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 3178628096 bytes 2026-02-20T22:29:54.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 3221054464 bytes 2026-02-20T22:29:54.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 3270451200 bytes 2026-02-20T22:29:54.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3298975744 bytes 2026-02-20T22:29:54.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3302089728 bytes 2026-02-20T22:29:54.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3303248896 bytes 2026-02-20T22:29:54.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3367766016 bytes 2026-02-20T22:29:54.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3390727168 bytes 2026-02-20T22:29:54.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3446062080 bytes 2026-02-20T22:29:54.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3471927296 bytes 2026-02-20T22:29:54.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3507277824 bytes 2026-02-20T22:29:54.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3544153088 bytes 2026-02-20T22:29:54.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3653735424 bytes 2026-02-20T22:29:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3710541824 bytes 2026-02-20T22:29:55.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3767190528 bytes 2026-02-20T22:29:55.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3814324224 bytes 2026-02-20T22:29:55.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3849185280 bytes 2026-02-20T22:29:55.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3891150848 bytes 2026-02-20T22:29:55.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3991886848 bytes 2026-02-20T22:29:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 4076291072 bytes 2026-02-20T22:29:55.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 4145236992 bytes 2026-02-20T22:29:55.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 4205233152 bytes 2026-02-20T22:29:55.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 4324654080 bytes 2026-02-20T22:29:55.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 4355867648 bytes 2026-02-20T22:29:55.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4429407232 bytes 2026-02-20T22:29:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4492663808 bytes 2026-02-20T22:29:56.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4545464320 bytes 2026-02-20T22:29:56.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4578546688 bytes 2026-02-20T22:29:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4615966720 bytes 2026-02-20T22:29:56.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4617393152 bytes 2026-02-20T22:29:56.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4623856640 bytes 2026-02-20T22:29:56.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4664664064 bytes 2026-02-20T22:29:56.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4702090240 bytes 2026-02-20T22:29:56.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4762262528 bytes 2026-02-20T22:29:56.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4779658240 bytes 2026-02-20T22:29:56.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4794869760 bytes 2026-02-20T22:29:56.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4856150016 bytes 2026-02-20T22:29:56.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4927444992 bytes 2026-02-20T22:29:56.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4978181120 bytes 2026-02-20T22:29:56.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 5047097344 bytes 2026-02-20T22:29:56.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 5110327296 bytes 2026-02-20T22:29:56.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:29:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:29:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:29:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:29:57.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:29:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:29:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:29:57.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":17856716.8,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"} 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:29:57 2026-02-20T22:29:57.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:29:56' 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:29:57.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:29:57.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:29:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":17856716.8,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"} 2026-02-20T22:29:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:29:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:29:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:29:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:29:57.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:29:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:29:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:29:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:29:57' 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:29:57' 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:29:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:29:57' 2026-02-20T22:29:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:29:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:29:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:29:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":17856716.8,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:29:57cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:29:56' 2026-02-20T22:29:57.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:29:57.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:29:57.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:29:57' 2026-02-20T22:29:57.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:29:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:29:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:29:57.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:29:57.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:29:57.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:57.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:29:57.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap5 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap5 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5 2026-02-20T22:29:57.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:30:27.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:27.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:30:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:30:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:30:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:30:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:30:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:28.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:30:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:30:28.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:30:28.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:30:28.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:30:28.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:30:28.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:30:28.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:30:28.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:30:28.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:30:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:30:28.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:30:28.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:30:28.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:30:28.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:30:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:30:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:30:28.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:30:28.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:30:28.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:30:30.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 17 2026-02-20T22:30:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:30:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:30:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:30.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:30:30.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:30:30.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:30.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:30:30.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:30:30.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:30:30.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:30:30.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:30:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:30:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:30:30.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:30:30.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:30:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:30:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=17 2026-02-20T22:30:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:30:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:30:30.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:30:31.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:30:31.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:30:31.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:30:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:30:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:30:31.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:30:31.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:30:31.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 17 = '' 2026-02-20T22:30:31.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:30:31.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:30:31.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:30:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:30:31.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:30:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:30:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:30:31.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:30:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:30:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 17 = '' 2026-02-20T22:30:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:30:31.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:30:32.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4057035080317225) > weight (0.25) 2026-02-20T22:30:32.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.5783552319786437) > weight (0.25) 2026-02-20T22:30:32.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.3450364820354854) > weight (0.25) 2026-02-20T22:30:32.031 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.5243158789520286) > weight (0.25) 2026-02-20T22:30:32.031 INFO:tasks.rbd_mirror_thrash:waiting for 115.10654112850996 secs before thrashing 2026-02-20T22:30:32.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:32.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:32.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:32.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:32.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:32.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:30:32.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:30:32.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:30:32.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:30:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:30:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:30:32.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:30:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:30:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 17 = '' 2026-02-20T22:30:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:30:32.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:30:34.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:34.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:34.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:34.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:34.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:30:34.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:30:34.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:30:34.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=17 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 17 = 17 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:30:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:30:35.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2026-02-20T22:30:35.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 10 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:30:35.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:29:48 2026 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:30:35.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:30:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:35.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap4-clean ']' 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap4-clean 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4-clean 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:35.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:35.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:30:35.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:30:35.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:36.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:30:36.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:36.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap5 2026-02-20T22:30:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:36.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:36.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:30:36.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:30:36.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:37.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:30:37.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:37.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:30:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap5-clean 2026-02-20T22:30:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap5-clean 2026-02-20T22:30:37.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:37.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:37.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:37.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5-clean 2026-02-20T22:30:37.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:30:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:30:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:30:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:37.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:37.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:30:37.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=11 2026-02-20T22:30:37.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:30:37.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:30:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 28623872 bytes 2026-02-20T22:30:38.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 45879296 bytes 2026-02-20T22:30:38.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 69042176 bytes 2026-02-20T22:30:38.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 132970496 bytes 2026-02-20T22:30:38.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 169171968 bytes 2026-02-20T22:30:38.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 196555776 bytes 2026-02-20T22:30:38.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 251071488 bytes 2026-02-20T22:30:38.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 284673024 bytes 2026-02-20T22:30:38.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 331533312 bytes 2026-02-20T22:30:38.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 390001664 bytes 2026-02-20T22:30:38.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 428173312 bytes 2026-02-20T22:30:38.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 465646592 bytes 2026-02-20T22:30:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 492946432 bytes 2026-02-20T22:30:38.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 527924224 bytes 2026-02-20T22:30:38.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 545939456 bytes 2026-02-20T22:30:38.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 653091840 bytes 2026-02-20T22:30:38.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 698504192 bytes 2026-02-20T22:30:38.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 726393856 bytes 2026-02-20T22:30:38.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 769920000 bytes 2026-02-20T22:30:38.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 802545664 bytes 2026-02-20T22:30:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 835257344 bytes 2026-02-20T22:30:38.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 863539200 bytes 2026-02-20T22:30:39.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 945148928 bytes 2026-02-20T22:30:39.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1018092544 bytes 2026-02-20T22:30:39.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1063610368 bytes 2026-02-20T22:30:39.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1093964800 bytes 2026-02-20T22:30:39.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1116877824 bytes 2026-02-20T22:30:39.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1163288576 bytes 2026-02-20T22:30:39.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1180653568 bytes 2026-02-20T22:30:39.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1242725376 bytes 2026-02-20T22:30:39.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1306048512 bytes 2026-02-20T22:30:39.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1331667968 bytes 2026-02-20T22:30:39.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1390981120 bytes 2026-02-20T22:30:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1403259904 bytes 2026-02-20T22:30:39.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1458153472 bytes 2026-02-20T22:30:39.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1477812224 bytes 2026-02-20T22:30:39.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1583350784 bytes 2026-02-20T22:30:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1620805632 bytes 2026-02-20T22:30:40.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1701641216 bytes 2026-02-20T22:30:40.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1704122368 bytes 2026-02-20T22:30:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1760934912 bytes 2026-02-20T22:30:40.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1775934464 bytes 2026-02-20T22:30:40.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1810363392 bytes 2026-02-20T22:30:40.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1830676480 bytes 2026-02-20T22:30:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1866481664 bytes 2026-02-20T22:30:40.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1909225472 bytes 2026-02-20T22:30:40.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1943234560 bytes 2026-02-20T22:30:40.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1970215936 bytes 2026-02-20T22:30:40.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2053827584 bytes 2026-02-20T22:30:40.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2103787520 bytes 2026-02-20T22:30:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2139706368 bytes 2026-02-20T22:30:40.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2148725760 bytes 2026-02-20T22:30:40.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2154225664 bytes 2026-02-20T22:30:40.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2202105856 bytes 2026-02-20T22:30:41.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2237261824 bytes 2026-02-20T22:30:41.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2305219584 bytes 2026-02-20T22:30:41.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2342275072 bytes 2026-02-20T22:30:41.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2361245696 bytes 2026-02-20T22:30:41.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2379674624 bytes 2026-02-20T22:30:41.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2395902976 bytes 2026-02-20T22:30:41.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2403240960 bytes 2026-02-20T22:30:41.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2512687104 bytes 2026-02-20T22:30:41.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2554523648 bytes 2026-02-20T22:30:41.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2626585600 bytes 2026-02-20T22:30:41.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2714802176 bytes 2026-02-20T22:30:41.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2763129856 bytes 2026-02-20T22:30:41.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2784638976 bytes 2026-02-20T22:30:41.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2821147648 bytes 2026-02-20T22:30:41.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2854471680 bytes 2026-02-20T22:30:41.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2906896384 bytes 2026-02-20T22:30:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2935998464 bytes 2026-02-20T22:30:41.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2974650368 bytes 2026-02-20T22:30:41.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3023129600 bytes 2026-02-20T22:30:42.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3128659968 bytes 2026-02-20T22:30:42.036 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:30:42.029+0000 7f7ed1951640 -1 reset not still connected to 0x55c7096a51e0 2026-02-20T22:30:42.036 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:30:42.029+0000 7f7ed1951640 -1 reset not still connected to 0x55c70ddc4270 2026-02-20T22:30:42.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3168227328 bytes 2026-02-20T22:30:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3208895488 bytes 2026-02-20T22:30:42.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3240421376 bytes 2026-02-20T22:30:42.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3283293184 bytes 2026-02-20T22:30:42.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3332507648 bytes 2026-02-20T22:30:42.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3346316288 bytes 2026-02-20T22:30:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3367887872 bytes 2026-02-20T22:30:42.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3405583360 bytes 2026-02-20T22:30:42.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3420821504 bytes 2026-02-20T22:30:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3428146176 bytes 2026-02-20T22:30:42.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3474778112 bytes 2026-02-20T22:30:42.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3510018048 bytes 2026-02-20T22:30:42.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3536729088 bytes 2026-02-20T22:30:42.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3584655360 bytes 2026-02-20T22:30:42.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3670848512 bytes 2026-02-20T22:30:43.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3716400128 bytes 2026-02-20T22:30:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3738914816 bytes 2026-02-20T22:30:43.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3791366144 bytes 2026-02-20T22:30:43.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3886705664 bytes 2026-02-20T22:30:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3903424512 bytes 2026-02-20T22:30:43.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3955034112 bytes 2026-02-20T22:30:43.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3957515264 bytes 2026-02-20T22:30:43.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4044034048 bytes 2026-02-20T22:30:43.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4067533824 bytes 2026-02-20T22:30:43.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4116780032 bytes 2026-02-20T22:30:43.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4123939840 bytes 2026-02-20T22:30:43.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4162204672 bytes 2026-02-20T22:30:43.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4220574720 bytes 2026-02-20T22:30:43.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4300717056 bytes 2026-02-20T22:30:43.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4335082496 bytes 2026-02-20T22:30:43.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4356676608 bytes 2026-02-20T22:30:43.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4377398272 bytes 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.575740840075544) > weight (0.25) 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5494055691498817) > weight (0.25) 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7033481446447918) > weight (0.25) 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2026-02-20T22:30:43.816 INFO:tasks.rbd_mirror_thrash:waiting for 0.5111645479240545 secs before reviving daemons 2026-02-20T22:30:43.817 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:30:43.809+0000 7f60059d2640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 22971) UID: 1000 2026-02-20T22:30:43.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4401401856 bytes 2026-02-20T22:30:43.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4429671424 bytes 2026-02-20T22:30:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4501857280 bytes 2026-02-20T22:30:44.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4573850624 bytes 2026-02-20T22:30:44.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4609069056 bytes 2026-02-20T22:30:44.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4639526912 bytes 2026-02-20T22:30:44.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4656338944 bytes 2026-02-20T22:30:44.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4696434688 bytes 2026-02-20T22:30:44.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4772155392 bytes 2026-02-20T22:30:44.328 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2026-02-20T22:30:44.328 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:30:44.329 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2026-02-20T22:30:44.329 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-20T22:30:44.329 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-20T22:30:44.331 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-20T22:30:44.331 INFO:tasks.rbd_mirror_thrash:waiting for 60.62967482459597 secs before thrashing 2026-02-20T22:30:44.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4792643584 bytes 2026-02-20T22:30:44.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4840622080 bytes 2026-02-20T22:30:44.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4917960704 bytes 2026-02-20T22:30:44.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5006046208 bytes 2026-02-20T22:30:44.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5031652352 bytes 2026-02-20T22:30:44.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5127335936 bytes 2026-02-20T22:30:44.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5157690368 bytes 2026-02-20T22:30:44.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5175864320 bytes 2026-02-20T22:30:44.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5232015360 bytes 2026-02-20T22:30:44.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5279479808 bytes 2026-02-20T22:30:44.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5310023680 bytes 2026-02-20T22:30:44.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5343817728 bytes 2026-02-20T22:30:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5344414720 bytes 2026-02-20T22:30:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5362953216 bytes 2026-02-20T22:30:45.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5417714688 bytes 2026-02-20T22:30:45.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5456919552 bytes 2026-02-20T22:30:45.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5481083904 bytes 2026-02-20T22:30:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5558431744 bytes 2026-02-20T22:30:45.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5637043200 bytes 2026-02-20T22:30:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5661421568 bytes 2026-02-20T22:30:45.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5713630208 bytes 2026-02-20T22:30:45.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5776079872 bytes 2026-02-20T22:30:45.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5822379008 bytes 2026-02-20T22:30:45.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5895702528 bytes 2026-02-20T22:30:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5927872512 bytes 2026-02-20T22:30:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5967090688 bytes 2026-02-20T22:30:45.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5987427328 bytes 2026-02-20T22:30:45.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6016975872 bytes 2026-02-20T22:30:45.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6018430976 bytes 2026-02-20T22:30:45.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6088067072 bytes 2026-02-20T22:30:45.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6133062656 bytes 2026-02-20T22:30:45.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6185377792 bytes 2026-02-20T22:30:46.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6213204992 bytes 2026-02-20T22:30:46.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6225052672 bytes 2026-02-20T22:30:46.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6256637952 bytes 2026-02-20T22:30:46.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6352209920 bytes 2026-02-20T22:30:46.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6409304064 bytes 2026-02-20T22:30:46.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6417035264 bytes 2026-02-20T22:30:46.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6461434880 bytes 2026-02-20T22:30:46.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6481687552 bytes 2026-02-20T22:30:46.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6508112896 bytes 2026-02-20T22:30:46.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6670577664 bytes 2026-02-20T22:30:46.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6688034816 bytes 2026-02-20T22:30:46.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6755345408 bytes 2026-02-20T22:30:46.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6784801792 bytes 2026-02-20T22:30:46.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6875167744 bytes 2026-02-20T22:30:46.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6901889024 bytes 2026-02-20T22:30:46.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6965088256 bytes 2026-02-20T22:30:46.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6970528768 bytes 2026-02-20T22:30:46.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6971157504 bytes 2026-02-20T22:30:46.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6978791424 bytes 2026-02-20T22:30:47.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7054663680 bytes 2026-02-20T22:30:47.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7055342592 bytes 2026-02-20T22:30:47.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7145796608 bytes 2026-02-20T22:30:47.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7227452416 bytes 2026-02-20T22:30:47.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7275849728 bytes 2026-02-20T22:30:47.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7338055680 bytes 2026-02-20T22:30:47.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7376628736 bytes 2026-02-20T22:30:47.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7410168832 bytes 2026-02-20T22:30:47.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7412128768 bytes 2026-02-20T22:30:47.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7509925888 bytes 2026-02-20T22:30:47.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7546018816 bytes 2026-02-20T22:30:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7599108096 bytes 2026-02-20T22:30:47.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7688657920 bytes 2026-02-20T22:30:47.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7741447168 bytes 2026-02-20T22:30:47.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7799860224 bytes 2026-02-20T22:30:47.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7859964928 bytes 2026-02-20T22:30:47.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7870098432 bytes 2026-02-20T22:30:48.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7935579136 bytes 2026-02-20T22:30:48.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7953232896 bytes 2026-02-20T22:30:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8039826432 bytes 2026-02-20T22:30:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8066543616 bytes 2026-02-20T22:30:48.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8103390208 bytes 2026-02-20T22:30:48.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8148675584 bytes 2026-02-20T22:30:48.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8164249600 bytes 2026-02-20T22:30:48.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8232074240 bytes 2026-02-20T22:30:48.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8254810112 bytes 2026-02-20T22:30:48.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8295095296 bytes 2026-02-20T22:30:48.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8328823808 bytes 2026-02-20T22:30:48.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8396090368 bytes 2026-02-20T22:30:48.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8463113216 bytes 2026-02-20T22:30:48.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:30:48.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:30:48.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:30:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:30:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:30:49.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"} 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:30:27 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:30:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:30:26' 2026-02-20T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:30:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:30:49.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:30:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"} 2026-02-20T22:30:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:30:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:30:49.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:30:49.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:30:49.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:30:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:30:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:30:49.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:30:27' 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:30:27' 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:49.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:30:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:30:27' 2026-02-20T22:30:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:30:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:30:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:30:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535859712.0,"last_snapshot_bytes":535701504,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626583,"remote_snapshot_timestamp":1771626583,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:30:27cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:30:26' 2026-02-20T22:30:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:30:49.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:30:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:30:27' 2026-02-20T22:30:49.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:30:49.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:30:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:30:49.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:30:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap6 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap6 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:30:49.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:30:49.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6 2026-02-20T22:30:49.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:31:19.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:19.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:31:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:31:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:31:19.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:31:20.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:31:20.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:31:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:31:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:31:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:31:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:31:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:31:20.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:31:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:31:20.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:31:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:31:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:31:20.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:31:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:31:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:31:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:31:20.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:31:20.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:31:20.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:31:20.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:31:20.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:31:20.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:20.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:31:22.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 20 2026-02-20T22:31:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:31:22.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:31:22.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:31:22.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:31:22.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:31:22.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:31:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:31:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:31:22.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:31:22.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:31:22.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=20 2026-02-20T22:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:31:22.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:31:23.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:31:23.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:31:23.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:31:23.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:31:23.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:31:23.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:31:23.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:31:23.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:31:23.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 20 = '' 2026-02-20T22:31:23.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:31:23.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:31:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:31:23.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:31:23.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:31:23.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:31:23.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:31:23.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:31:23.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:31:23.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 20 = '' 2026-02-20T22:31:23.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:31:23.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:31:24.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:24.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:24.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:31:24.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:31:24.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:31:24.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:31:24.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:31:24.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:31:24.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:31:24.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 20 = '' 2026-02-20T22:31:24.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:31:24.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:31:26.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:26.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:26.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:26.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:26.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:31:26.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:31:26.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:31:26.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=20 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 20 = 20 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:31:26.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:31:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2026-02-20T22:31:27.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:31:27.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:31:27.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 12 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:29:48 2026 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:31:27.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:31:27.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:31:27.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:31:27.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:31:27.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap5-clean ']' 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap5-clean 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:27.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5-clean 2026-02-20T22:31:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:31:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:31:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:27.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:31:27.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:31:27.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:31:28.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:28.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap6 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:31:28.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:28.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:31:28.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:31:28.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:29.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:31:29.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:29.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:31:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap6-clean 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap6-clean 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6-clean 2026-02-20T22:31:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:30.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:31:30.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:31:30.927 INFO:tasks.ceph.cluster2.osd.1.trial107.stderr:2026-02-20T22:31:30.920+0000 7fc7c5383640 -1 reset not still connected to 0x564bfe068ea0 2026-02-20T22:31:30.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 28653568 bytes 2026-02-20T22:31:31.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 106660864 bytes 2026-02-20T22:31:31.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 139257856 bytes 2026-02-20T22:31:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 180296704 bytes 2026-02-20T22:31:31.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 203526144 bytes 2026-02-20T22:31:31.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 240193536 bytes 2026-02-20T22:31:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 297879552 bytes 2026-02-20T22:31:31.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 365008896 bytes 2026-02-20T22:31:31.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 416292864 bytes 2026-02-20T22:31:31.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 485017600 bytes 2026-02-20T22:31:31.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 538592256 bytes 2026-02-20T22:31:31.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 561231872 bytes 2026-02-20T22:31:31.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 589046784 bytes 2026-02-20T22:31:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 636047360 bytes 2026-02-20T22:31:31.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 702846976 bytes 2026-02-20T22:31:31.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 732829696 bytes 2026-02-20T22:31:31.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 747230208 bytes 2026-02-20T22:31:31.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 786861056 bytes 2026-02-20T22:31:31.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 885946368 bytes 2026-02-20T22:31:32.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 949023744 bytes 2026-02-20T22:31:32.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 955742208 bytes 2026-02-20T22:31:32.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 972178432 bytes 2026-02-20T22:31:32.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1031576576 bytes 2026-02-20T22:31:32.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1113996288 bytes 2026-02-20T22:31:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1192621056 bytes 2026-02-20T22:31:32.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1242120192 bytes 2026-02-20T22:31:32.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1288740864 bytes 2026-02-20T22:31:32.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1315934208 bytes 2026-02-20T22:31:32.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1362982912 bytes 2026-02-20T22:31:32.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1409695744 bytes 2026-02-20T22:31:32.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1411235840 bytes 2026-02-20T22:31:32.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1471704064 bytes 2026-02-20T22:31:32.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1513420800 bytes 2026-02-20T22:31:32.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1573612544 bytes 2026-02-20T22:31:32.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1641355264 bytes 2026-02-20T22:31:32.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1651095552 bytes 2026-02-20T22:31:32.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1679888384 bytes 2026-02-20T22:31:32.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1699977216 bytes 2026-02-20T22:31:32.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1743598592 bytes 2026-02-20T22:31:33.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1794711552 bytes 2026-02-20T22:31:33.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1847566336 bytes 2026-02-20T22:31:33.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1887777792 bytes 2026-02-20T22:31:33.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1964670976 bytes 2026-02-20T22:31:33.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2043036672 bytes 2026-02-20T22:31:33.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2113891328 bytes 2026-02-20T22:31:33.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2181856256 bytes 2026-02-20T22:31:33.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2220125184 bytes 2026-02-20T22:31:33.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2347102208 bytes 2026-02-20T22:31:33.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2396878848 bytes 2026-02-20T22:31:33.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2437684224 bytes 2026-02-20T22:31:33.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2495734784 bytes 2026-02-20T22:31:33.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2505241600 bytes 2026-02-20T22:31:33.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2547655680 bytes 2026-02-20T22:31:33.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2603880448 bytes 2026-02-20T22:31:33.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2650091520 bytes 2026-02-20T22:31:33.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2660819968 bytes 2026-02-20T22:31:34.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2726951936 bytes 2026-02-20T22:31:34.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2757019648 bytes 2026-02-20T22:31:34.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2792748032 bytes 2026-02-20T22:31:34.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2827097088 bytes 2026-02-20T22:31:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2881734656 bytes 2026-02-20T22:31:34.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2904521728 bytes 2026-02-20T22:31:34.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2942338048 bytes 2026-02-20T22:31:34.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2966241280 bytes 2026-02-20T22:31:34.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2966847488 bytes 2026-02-20T22:31:34.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3019512832 bytes 2026-02-20T22:31:34.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3043543040 bytes 2026-02-20T22:31:34.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3044167680 bytes 2026-02-20T22:31:34.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3072594944 bytes 2026-02-20T22:31:34.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3092950016 bytes 2026-02-20T22:31:34.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3142466560 bytes 2026-02-20T22:31:34.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3211499520 bytes 2026-02-20T22:31:34.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3263369216 bytes 2026-02-20T22:31:34.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3300825088 bytes 2026-02-20T22:31:34.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3372630016 bytes 2026-02-20T22:31:34.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3423778816 bytes 2026-02-20T22:31:34.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3426099200 bytes 2026-02-20T22:31:35.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3458814976 bytes 2026-02-20T22:31:35.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3494923264 bytes 2026-02-20T22:31:35.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3501933568 bytes 2026-02-20T22:31:35.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3560546304 bytes 2026-02-20T22:31:35.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3644123136 bytes 2026-02-20T22:31:35.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3681699840 bytes 2026-02-20T22:31:35.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3715825664 bytes 2026-02-20T22:31:35.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3761005568 bytes 2026-02-20T22:31:35.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3781677056 bytes 2026-02-20T22:31:35.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3798326272 bytes 2026-02-20T22:31:35.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3853066240 bytes 2026-02-20T22:31:35.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3886166016 bytes 2026-02-20T22:31:35.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3944307712 bytes 2026-02-20T22:31:35.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3985016832 bytes 2026-02-20T22:31:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3985641472 bytes 2026-02-20T22:31:35.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4000632832 bytes 2026-02-20T22:31:35.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4041207808 bytes 2026-02-20T22:31:35.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4059642880 bytes 2026-02-20T22:31:35.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4125747200 bytes 2026-02-20T22:31:35.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4162205696 bytes 2026-02-20T22:31:36.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4253507584 bytes 2026-02-20T22:31:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4300830720 bytes 2026-02-20T22:31:36.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4324812800 bytes 2026-02-20T22:31:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4351779840 bytes 2026-02-20T22:31:36.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4379057152 bytes 2026-02-20T22:31:36.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4403102720 bytes 2026-02-20T22:31:36.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4428758016 bytes 2026-02-20T22:31:36.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4530338816 bytes 2026-02-20T22:31:36.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4562142208 bytes 2026-02-20T22:31:36.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4578865152 bytes 2026-02-20T22:31:36.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4608368640 bytes 2026-02-20T22:31:36.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4678704128 bytes 2026-02-20T22:31:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4730108928 bytes 2026-02-20T22:31:36.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4739334144 bytes 2026-02-20T22:31:36.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4819645440 bytes 2026-02-20T22:31:36.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4855191552 bytes 2026-02-20T22:31:36.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4877743104 bytes 2026-02-20T22:31:36.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4917515264 bytes 2026-02-20T22:31:36.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4941053952 bytes 2026-02-20T22:31:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5015860224 bytes 2026-02-20T22:31:37.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5033666560 bytes 2026-02-20T22:31:37.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5047636992 bytes 2026-02-20T22:31:37.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5063778304 bytes 2026-02-20T22:31:37.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5120488448 bytes 2026-02-20T22:31:37.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5147486208 bytes 2026-02-20T22:31:37.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5177133056 bytes 2026-02-20T22:31:37.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5214130176 bytes 2026-02-20T22:31:37.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5270644736 bytes 2026-02-20T22:31:37.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5308244992 bytes 2026-02-20T22:31:37.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5322315776 bytes 2026-02-20T22:31:37.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5345034240 bytes 2026-02-20T22:31:37.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5381834752 bytes 2026-02-20T22:31:37.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5437125632 bytes 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:31:37.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:31:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:31:38.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:31:38.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":16078165.33,"bytes_per_snapshot":534672384.0,"last_snapshot_bytes":536335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626681,"remote_snapshot_timestamp":1771626681,"replay_state":"idle"} 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:31:27 2026-02-20T22:31:38.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:31:26' 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:31:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:31:38.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:31:38.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":16078165.33,"bytes_per_snapshot":534672384.0,"last_snapshot_bytes":536335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626681,"remote_snapshot_timestamp":1771626681,"replay_state":"idle"} 2026-02-20T22:31:38.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:31:38.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:31:38.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:31:38.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:31:38.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:31:38.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:31:38.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:31:38.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:31:27' 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:31:27' 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:31:38.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:31:27' 2026-02-20T22:31:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:31:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:31:38.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:31:38.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42124265mirror.1trial10720.2.0-677-gf7870454falseOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":16078165.33,"bytes_per_snapshot":534672384.0,"last_snapshot_bytes":536335360,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626681,"remote_snapshot_timestamp":1771626681,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:31:27cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:31:26' 2026-02-20T22:31:38.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:31:38.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:31:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:31:27' 2026-02-20T22:31:38.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:31:38.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:31:38.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:31:38.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:31:38.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:31:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:31:38.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap7 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap7 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7 2026-02-20T22:31:38.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:31:44.962 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:31:44.963 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:31:44.963 INFO:tasks.rbd_mirror_thrash:waiting for 6.521597418550172 secs before reviving daemons 2026-02-20T22:31:44.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:31:44.956+0000 7feaa120a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 19583) UID: 1000 2026-02-20T22:31:51.485 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:31:51.486 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:31:51.486 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:31:51.486 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:31:51.486 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:31:51.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:31:51.489 INFO:tasks.rbd_mirror_thrash:waiting for 114.54064147687748 secs before thrashing 2026-02-20T22:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:07.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:32:07.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:32:08.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:32:08.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:32:08.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:32:08.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:32:08.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:32:08.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:32:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:32:08.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:32:08.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:32:08.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:32:08.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:32:08.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:08.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:08.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:08.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:32:08.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:32:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:32:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:32:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:32:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:32:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:32:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:08.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:08.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:08.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:32:10.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 23 2026-02-20T22:32:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:32:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:32:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:32:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:32:10.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:10.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:32:10.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:32:10.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:32:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:32:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:32:10.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:32:10.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:32:10.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:32:11.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:32:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=23 2026-02-20T22:32:11.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:32:11.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:32:11.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:32:11.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:11.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:11.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:11.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:11.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:11.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:32:11.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:32:11.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:32:11.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:32:11.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:32:11.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:32:11.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:32:11.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:32:11.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 23 = '' 2026-02-20T22:32:11.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:32:11.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:32:11.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:32:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:32:11.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:32:11.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:32:11.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:32:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:32:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:32:11.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 23 = '' 2026-02-20T22:32:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:32:11.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:32:11.950 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:32:11.941+0000 7fa6ec82b640 -1 reset not still connected to 0x555b451daa90 2026-02-20T22:32:11.950 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:32:11.941+0000 7fa6ec82b640 -1 reset not still connected to 0x555b451db450 2026-02-20T22:32:11.950 INFO:tasks.ceph.cluster1.osd.1.trial032.stderr:2026-02-20T22:32:11.941+0000 7fa6ec82b640 -1 reset not still connected to 0x555b45c56820 2026-02-20T22:32:12.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:12.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:12.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:12.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:12.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:32:12.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:32:12.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:32:12.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:32:12.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=23 2026-02-20T22:32:12.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 23 = 23 2026-02-20T22:32:12.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:32:12.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:32:13.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2026-02-20T22:32:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 14 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:32:13.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:31:27 2026 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:32:13.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap6-clean ']' 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap6-clean 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6-clean 2026-02-20T22:32:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:32:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:32:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:32:14.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:32:14.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:14.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:32:14.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:15.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap7 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:15.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:32:15.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:32:15.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:16.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:32:16.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:16.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:32:16.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap7-clean 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap7-clean 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7-clean 2026-02-20T22:32:16.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:32:17.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:32:17.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:32:17.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:17.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:17.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:17.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:32:17.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:32:17.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:32:17.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:32:17.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 59926528 bytes 2026-02-20T22:32:17.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 76223488 bytes 2026-02-20T22:32:17.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 107783168 bytes 2026-02-20T22:32:17.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 108384256 bytes 2026-02-20T22:32:17.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 159891456 bytes 2026-02-20T22:32:17.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 219784192 bytes 2026-02-20T22:32:17.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 225975296 bytes 2026-02-20T22:32:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 291803136 bytes 2026-02-20T22:32:18.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 317965312 bytes 2026-02-20T22:32:18.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 346398720 bytes 2026-02-20T22:32:18.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 347021312 bytes 2026-02-20T22:32:18.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 408802304 bytes 2026-02-20T22:32:18.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 418294784 bytes 2026-02-20T22:32:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 446211072 bytes 2026-02-20T22:32:18.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 498737152 bytes 2026-02-20T22:32:18.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 526611456 bytes 2026-02-20T22:32:18.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 605369344 bytes 2026-02-20T22:32:18.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 668158976 bytes 2026-02-20T22:32:18.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 676220928 bytes 2026-02-20T22:32:18.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 693439488 bytes 2026-02-20T22:32:18.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 741425152 bytes 2026-02-20T22:32:18.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 776169472 bytes 2026-02-20T22:32:18.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 840758272 bytes 2026-02-20T22:32:18.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 872440832 bytes 2026-02-20T22:32:18.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 920211456 bytes 2026-02-20T22:32:18.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 962059264 bytes 2026-02-20T22:32:18.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 984034304 bytes 2026-02-20T22:32:19.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1019558912 bytes 2026-02-20T22:32:19.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1025995776 bytes 2026-02-20T22:32:19.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1026612224 bytes 2026-02-20T22:32:19.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1085256704 bytes 2026-02-20T22:32:19.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1155795968 bytes 2026-02-20T22:32:19.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1187464192 bytes 2026-02-20T22:32:19.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1306535936 bytes 2026-02-20T22:32:19.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1332517888 bytes 2026-02-20T22:32:19.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1343020032 bytes 2026-02-20T22:32:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1365599232 bytes 2026-02-20T22:32:19.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1386745856 bytes 2026-02-20T22:32:19.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1402522624 bytes 2026-02-20T22:32:19.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1428126720 bytes 2026-02-20T22:32:19.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1433809920 bytes 2026-02-20T22:32:19.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1463729152 bytes 2026-02-20T22:32:19.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1502593024 bytes 2026-02-20T22:32:19.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1547146240 bytes 2026-02-20T22:32:19.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1600499712 bytes 2026-02-20T22:32:19.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1629075456 bytes 2026-02-20T22:32:19.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1640261632 bytes 2026-02-20T22:32:19.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1643246592 bytes 2026-02-20T22:32:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1643859968 bytes 2026-02-20T22:32:20.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1667318784 bytes 2026-02-20T22:32:20.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1723051008 bytes 2026-02-20T22:32:20.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1776295936 bytes 2026-02-20T22:32:20.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1828912128 bytes 2026-02-20T22:32:20.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1864151040 bytes 2026-02-20T22:32:20.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 1915997184 bytes 2026-02-20T22:32:20.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 1928478720 bytes 2026-02-20T22:32:20.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 1979518976 bytes 2026-02-20T22:32:20.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2013732864 bytes 2026-02-20T22:32:20.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2061357056 bytes 2026-02-20T22:32:20.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2114576384 bytes 2026-02-20T22:32:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2157794304 bytes 2026-02-20T22:32:20.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2192158720 bytes 2026-02-20T22:32:20.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2196791296 bytes 2026-02-20T22:32:20.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2202925056 bytes 2026-02-20T22:32:20.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2268420096 bytes 2026-02-20T22:32:20.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2318823424 bytes 2026-02-20T22:32:20.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2393909248 bytes 2026-02-20T22:32:21.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2477865984 bytes 2026-02-20T22:32:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2498702336 bytes 2026-02-20T22:32:21.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2540524544 bytes 2026-02-20T22:32:21.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2547848192 bytes 2026-02-20T22:32:21.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2566638592 bytes 2026-02-20T22:32:21.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2612218880 bytes 2026-02-20T22:32:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2622575616 bytes 2026-02-20T22:32:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2675709952 bytes 2026-02-20T22:32:21.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2681295872 bytes 2026-02-20T22:32:21.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2748393472 bytes 2026-02-20T22:32:21.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2748989440 bytes 2026-02-20T22:32:21.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2782828544 bytes 2026-02-20T22:32:21.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 2832288768 bytes 2026-02-20T22:32:21.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 2835444736 bytes 2026-02-20T22:32:21.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 2851926016 bytes 2026-02-20T22:32:21.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 2905503744 bytes 2026-02-20T22:32:21.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 2926703616 bytes 2026-02-20T22:32:21.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 2985009152 bytes 2026-02-20T22:32:21.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3000360960 bytes 2026-02-20T22:32:21.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3057101824 bytes 2026-02-20T22:32:21.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3071473664 bytes 2026-02-20T22:32:21.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3130934272 bytes 2026-02-20T22:32:22.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3171706880 bytes 2026-02-20T22:32:22.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3267357696 bytes 2026-02-20T22:32:22.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3288535040 bytes 2026-02-20T22:32:22.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3370748928 bytes 2026-02-20T22:32:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3441045504 bytes 2026-02-20T22:32:22.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3500486656 bytes 2026-02-20T22:32:22.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3581527040 bytes 2026-02-20T22:32:22.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3611106304 bytes 2026-02-20T22:32:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3617989632 bytes 2026-02-20T22:32:22.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3659645952 bytes 2026-02-20T22:32:22.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3750818816 bytes 2026-02-20T22:32:22.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3805135872 bytes 2026-02-20T22:32:22.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 3808673792 bytes 2026-02-20T22:32:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 3834493952 bytes 2026-02-20T22:32:22.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 3859722240 bytes 2026-02-20T22:32:22.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 3897755648 bytes 2026-02-20T22:32:22.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 3946886144 bytes 2026-02-20T22:32:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 3947484160 bytes 2026-02-20T22:32:23.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 3963337728 bytes 2026-02-20T22:32:23.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 3973232640 bytes 2026-02-20T22:32:23.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4035232768 bytes 2026-02-20T22:32:23.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4119188480 bytes 2026-02-20T22:32:23.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4152102912 bytes 2026-02-20T22:32:23.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4204359680 bytes 2026-02-20T22:32:23.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4255342592 bytes 2026-02-20T22:32:23.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4297997312 bytes 2026-02-20T22:32:23.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4356060160 bytes 2026-02-20T22:32:23.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4379309056 bytes 2026-02-20T22:32:23.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4420043776 bytes 2026-02-20T22:32:23.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4452330496 bytes 2026-02-20T22:32:23.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4495001600 bytes 2026-02-20T22:32:23.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4495628288 bytes 2026-02-20T22:32:23.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4600748032 bytes 2026-02-20T22:32:23.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4642026496 bytes 2026-02-20T22:32:23.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4676445184 bytes 2026-02-20T22:32:23.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4687839232 bytes 2026-02-20T22:32:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4710051840 bytes 2026-02-20T22:32:24.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 4738025472 bytes 2026-02-20T22:32:24.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 4794387456 bytes 2026-02-20T22:32:24.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 4908687360 bytes 2026-02-20T22:32:24.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 4952631296 bytes 2026-02-20T22:32:24.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 4972942336 bytes 2026-02-20T22:32:24.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 4995362816 bytes 2026-02-20T22:32:24.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 4998815744 bytes 2026-02-20T22:32:24.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5026184192 bytes 2026-02-20T22:32:24.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5073792000 bytes 2026-02-20T22:32:24.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5086597120 bytes 2026-02-20T22:32:24.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5106008064 bytes 2026-02-20T22:32:24.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5153973248 bytes 2026-02-20T22:32:24.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5247207424 bytes 2026-02-20T22:32:24.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5276990464 bytes 2026-02-20T22:32:24.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5356938240 bytes 2026-02-20T22:32:24.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5389363200 bytes 2026-02-20T22:32:24.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5405907968 bytes 2026-02-20T22:32:24.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5471235072 bytes 2026-02-20T22:32:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5487095808 bytes 2026-02-20T22:32:25.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5570579456 bytes 2026-02-20T22:32:25.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5571643392 bytes 2026-02-20T22:32:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5621087232 bytes 2026-02-20T22:32:25.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 5659962368 bytes 2026-02-20T22:32:25.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 5660565504 bytes 2026-02-20T22:32:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 5728303104 bytes 2026-02-20T22:32:25.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 5762438144 bytes 2026-02-20T22:32:25.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 5817050112 bytes 2026-02-20T22:32:25.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 5846726656 bytes 2026-02-20T22:32:25.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 5874001920 bytes 2026-02-20T22:32:25.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 5876008960 bytes 2026-02-20T22:32:25.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 5921120256 bytes 2026-02-20T22:32:25.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 5986896896 bytes 2026-02-20T22:32:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 5987519488 bytes 2026-02-20T22:32:25.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6051976192 bytes 2026-02-20T22:32:25.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6071579648 bytes 2026-02-20T22:32:25.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6119063552 bytes 2026-02-20T22:32:25.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6186358784 bytes 2026-02-20T22:32:25.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6225390592 bytes 2026-02-20T22:32:25.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6247525376 bytes 2026-02-20T22:32:26.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6281881600 bytes 2026-02-20T22:32:26.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6304306176 bytes 2026-02-20T22:32:26.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6330660864 bytes 2026-02-20T22:32:26.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6336375808 bytes 2026-02-20T22:32:26.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 6376899584 bytes 2026-02-20T22:32:26.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 6444833792 bytes 2026-02-20T22:32:26.335 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 6481803264 bytes 2026-02-20T22:32:26.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 6493441024 bytes 2026-02-20T22:32:26.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 6510522368 bytes 2026-02-20T22:32:26.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 6512131072 bytes 2026-02-20T22:32:26.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 6571722752 bytes 2026-02-20T22:32:26.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 6608219136 bytes 2026-02-20T22:32:26.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 6698054656 bytes 2026-02-20T22:32:26.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 6698653696 bytes 2026-02-20T22:32:26.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 6717615104 bytes 2026-02-20T22:32:26.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 6757072896 bytes 2026-02-20T22:32:26.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 6787342336 bytes 2026-02-20T22:32:26.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 6829908992 bytes 2026-02-20T22:32:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 6870598656 bytes 2026-02-20T22:32:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 6975152128 bytes 2026-02-20T22:32:27.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7021037568 bytes 2026-02-20T22:32:27.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7021967360 bytes 2026-02-20T22:32:27.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7062995968 bytes 2026-02-20T22:32:27.138 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8903499504664085) > weight (0.25) 2026-02-20T22:32:27.138 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-20T22:32:27.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-20T22:32:27.138 INFO:tasks.rbd_mirror_thrash:waiting for 8.395876622540747 secs before reviving daemons 2026-02-20T22:32:27.140 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T22:32:27.135+0000 7fb824baf640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 19529) UID: 1000 2026-02-20T22:32:27.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7123565568 bytes 2026-02-20T22:32:27.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7187909632 bytes 2026-02-20T22:32:27.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7241975808 bytes 2026-02-20T22:32:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7276746752 bytes 2026-02-20T22:32:27.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7287717888 bytes 2026-02-20T22:32:27.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7305467904 bytes 2026-02-20T22:32:27.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 7346167808 bytes 2026-02-20T22:32:27.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 7412082688 bytes 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:32:27.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:32:28.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:32:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":17843302.4,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"} 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:32:27 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:32:27' 2026-02-20T22:32:28.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:32:28.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:32:28.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:32:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":17843302.4,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"} 2026-02-20T22:32:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:32:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:32:28.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:32:28.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:32:28.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:32:28.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:32:28.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:32:28.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:32:27' 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:32:27' 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:32:27' 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:32:28.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:32:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='UNKNOWNUNKNOWNOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK4212mirror.1trial10720.2.0-677-gf7870454falseUNKNOWN42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":17843302.4,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:32:27cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:32:27' 2026-02-20T22:32:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:32:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:32:28.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:32:27' 2026-02-20T22:32:28.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:32:28.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:32:28.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:32:28.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap8 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap8 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8 2026-02-20T22:32:28.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:32:35.536 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-20T22:32:35.536 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:32:35.536 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-20T22:32:35.536 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-20T22:32:35.536 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-20T22:32:35.537 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-20T22:32:35.537 INFO:tasks.rbd_mirror_thrash:waiting for 78.14149245283974 secs before thrashing 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:32:58.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:32:58.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:32:59.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:32:59.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:32:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:32:59.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:32:59.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:32:59.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:32:59.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:32:59.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:32:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:32:59.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:59.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:59.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:32:59.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:32:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:32:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:32:59.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:32:59.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:32:59.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:32:59.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:32:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:32:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:32:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:32:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:32:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:33:01.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 26 2026-02-20T22:33:01.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:33:01.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:01.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:33:01.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:33:01.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:33:01.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:33:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:33:01.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:33:01.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:01.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:33:01.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:01.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:33:01.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=26 2026-02-20T22:33:01.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:01.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:33:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:02.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:02.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:02.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:02.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:02.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 26 = '' 2026-02-20T22:33:02.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:02.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:02.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:02.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:02.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:02.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 26 = '' 2026-02-20T22:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:33:03.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:03.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:03.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:03.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:03.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:03.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:03.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:03.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 26 = '' 2026-02-20T22:33:03.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:03.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:33:05.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:05.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:05.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:05.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:05.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=26 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 26 = 26 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:33:05.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:33:06.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2026-02-20T22:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 16 2026-02-20T22:33:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:31:27 2026 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:33:06.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:33:06.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:33:06.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:33:06.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:06.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap7-clean ']' 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap7-clean 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7-clean 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:06.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:06.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:33:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:07.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:33:07.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:07.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:07.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap8 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:07.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:07.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:33:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:08.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:33:08.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:08.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap8-clean 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap8-clean 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8-clean 2026-02-20T22:33:08.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:09.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:33:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=12 2026-02-20T22:33:09.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:33:09.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:33:09.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 41165824 bytes 2026-02-20T22:33:10.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 131981312 bytes 2026-02-20T22:33:10.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 132614144 bytes 2026-02-20T22:33:10.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 148609024 bytes 2026-02-20T22:33:10.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 239697920 bytes 2026-02-20T22:33:10.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 290039808 bytes 2026-02-20T22:33:10.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 358663168 bytes 2026-02-20T22:33:10.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 401654784 bytes 2026-02-20T22:33:10.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 455755776 bytes 2026-02-20T22:33:10.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 478669824 bytes 2026-02-20T22:33:10.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 491986944 bytes 2026-02-20T22:33:10.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 492585984 bytes 2026-02-20T22:33:10.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 526676992 bytes 2026-02-20T22:33:10.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 535252992 bytes 2026-02-20T22:33:10.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 573527040 bytes 2026-02-20T22:33:10.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 579870720 bytes 2026-02-20T22:33:10.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 622654464 bytes 2026-02-20T22:33:10.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 635375616 bytes 2026-02-20T22:33:10.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 662490112 bytes 2026-02-20T22:33:10.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 726652928 bytes 2026-02-20T22:33:10.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 773171200 bytes 2026-02-20T22:33:11.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 801731584 bytes 2026-02-20T22:33:11.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 831432704 bytes 2026-02-20T22:33:11.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 873618432 bytes 2026-02-20T22:33:11.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 894330880 bytes 2026-02-20T22:33:11.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 932455424 bytes 2026-02-20T22:33:11.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 946594816 bytes 2026-02-20T22:33:11.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 967565312 bytes 2026-02-20T22:33:11.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1023630336 bytes 2026-02-20T22:33:11.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1054479360 bytes 2026-02-20T22:33:11.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1091414016 bytes 2026-02-20T22:33:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1115375616 bytes 2026-02-20T22:33:11.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1159690240 bytes 2026-02-20T22:33:11.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1185478656 bytes 2026-02-20T22:33:11.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1229063168 bytes 2026-02-20T22:33:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1267848192 bytes 2026-02-20T22:33:11.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1277271040 bytes 2026-02-20T22:33:11.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1330564096 bytes 2026-02-20T22:33:11.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1377011712 bytes 2026-02-20T22:33:11.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1417625600 bytes 2026-02-20T22:33:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1427409920 bytes 2026-02-20T22:33:11.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1454638080 bytes 2026-02-20T22:33:12.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1498893312 bytes 2026-02-20T22:33:12.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1539536896 bytes 2026-02-20T22:33:12.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1569093632 bytes 2026-02-20T22:33:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1609624576 bytes 2026-02-20T22:33:12.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1612491776 bytes 2026-02-20T22:33:12.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1623904256 bytes 2026-02-20T22:33:12.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1633591296 bytes 2026-02-20T22:33:12.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1686650880 bytes 2026-02-20T22:33:12.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1716657152 bytes 2026-02-20T22:33:12.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1731747840 bytes 2026-02-20T22:33:12.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1755397120 bytes 2026-02-20T22:33:12.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1795115008 bytes 2026-02-20T22:33:12.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 1868128256 bytes 2026-02-20T22:33:12.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 1868721152 bytes 2026-02-20T22:33:12.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 1917616128 bytes 2026-02-20T22:33:12.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 1962082304 bytes 2026-02-20T22:33:12.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2011039744 bytes 2026-02-20T22:33:12.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2083859456 bytes 2026-02-20T22:33:12.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2103433216 bytes 2026-02-20T22:33:12.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2172251136 bytes 2026-02-20T22:33:13.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2182781952 bytes 2026-02-20T22:33:13.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2195245056 bytes 2026-02-20T22:33:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2216549376 bytes 2026-02-20T22:33:13.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2217155584 bytes 2026-02-20T22:33:13.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2263876608 bytes 2026-02-20T22:33:13.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2317961216 bytes 2026-02-20T22:33:13.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2376267776 bytes 2026-02-20T22:33:13.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2419823616 bytes 2026-02-20T22:33:13.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2455930880 bytes 2026-02-20T22:33:13.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2557443072 bytes 2026-02-20T22:33:13.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2603024384 bytes 2026-02-20T22:33:13.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2658590720 bytes 2026-02-20T22:33:13.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2723563520 bytes 2026-02-20T22:33:13.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2763043840 bytes 2026-02-20T22:33:13.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2842204160 bytes 2026-02-20T22:33:13.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2907373568 bytes 2026-02-20T22:33:13.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2948483072 bytes 2026-02-20T22:33:13.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3032496128 bytes 2026-02-20T22:33:14.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3130032128 bytes 2026-02-20T22:33:14.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3166160896 bytes 2026-02-20T22:33:14.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3198068736 bytes 2026-02-20T22:33:14.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3224985600 bytes 2026-02-20T22:33:14.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3249434624 bytes 2026-02-20T22:33:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3302991872 bytes 2026-02-20T22:33:14.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3385988096 bytes 2026-02-20T22:33:14.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3469626368 bytes 2026-02-20T22:33:14.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3512830976 bytes 2026-02-20T22:33:14.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3520747520 bytes 2026-02-20T22:33:14.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3603905536 bytes 2026-02-20T22:33:14.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3644150784 bytes 2026-02-20T22:33:14.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3671768064 bytes 2026-02-20T22:33:14.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3702251520 bytes 2026-02-20T22:33:14.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3726911488 bytes 2026-02-20T22:33:14.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3746514944 bytes 2026-02-20T22:33:14.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3797237760 bytes 2026-02-20T22:33:14.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3836152832 bytes 2026-02-20T22:33:15.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3879973888 bytes 2026-02-20T22:33:15.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3905845248 bytes 2026-02-20T22:33:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3983329280 bytes 2026-02-20T22:33:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4027837440 bytes 2026-02-20T22:33:15.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4050793472 bytes 2026-02-20T22:33:15.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4093184000 bytes 2026-02-20T22:33:15.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4093802496 bytes 2026-02-20T22:33:15.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4129739776 bytes 2026-02-20T22:33:15.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4214210560 bytes 2026-02-20T22:33:15.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4236788736 bytes 2026-02-20T22:33:15.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4277690368 bytes 2026-02-20T22:33:15.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4351770624 bytes 2026-02-20T22:33:15.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4384269312 bytes 2026-02-20T22:33:15.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4440309760 bytes 2026-02-20T22:33:15.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4481869824 bytes 2026-02-20T22:33:15.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4510674944 bytes 2026-02-20T22:33:15.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4511285248 bytes 2026-02-20T22:33:15.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4561818624 bytes 2026-02-20T22:33:15.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4585549824 bytes 2026-02-20T22:33:16.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4620403712 bytes 2026-02-20T22:33:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4675168256 bytes 2026-02-20T22:33:16.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4693090304 bytes 2026-02-20T22:33:16.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4717848576 bytes 2026-02-20T22:33:16.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4724751360 bytes 2026-02-20T22:33:16.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4763161600 bytes 2026-02-20T22:33:16.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4786669568 bytes 2026-02-20T22:33:16.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4820689920 bytes 2026-02-20T22:33:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4829092864 bytes 2026-02-20T22:33:16.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 4869636096 bytes 2026-02-20T22:33:16.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 4902905856 bytes 2026-02-20T22:33:16.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 4921731072 bytes 2026-02-20T22:33:16.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 4963401728 bytes 2026-02-20T22:33:16.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 4969467904 bytes 2026-02-20T22:33:16.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5038148608 bytes 2026-02-20T22:33:16.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5092781056 bytes 2026-02-20T22:33:16.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5168427008 bytes 2026-02-20T22:33:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5203003392 bytes 2026-02-20T22:33:16.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5261489152 bytes 2026-02-20T22:33:16.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5339636736 bytes 2026-02-20T22:33:17.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5379936256 bytes 2026-02-20T22:33:17.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5501034496 bytes 2026-02-20T22:33:17.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5561947136 bytes 2026-02-20T22:33:17.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5604740096 bytes 2026-02-20T22:33:17.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5660915712 bytes 2026-02-20T22:33:17.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5712814080 bytes 2026-02-20T22:33:17.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5761665024 bytes 2026-02-20T22:33:17.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5784218624 bytes 2026-02-20T22:33:17.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5861021696 bytes 2026-02-20T22:33:17.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5861604352 bytes 2026-02-20T22:33:17.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5921739776 bytes 2026-02-20T22:33:17.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 5976371200 bytes 2026-02-20T22:33:17.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6054932480 bytes 2026-02-20T22:33:17.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6055528448 bytes 2026-02-20T22:33:17.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6076624896 bytes 2026-02-20T22:33:17.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6127279104 bytes 2026-02-20T22:33:17.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6149626880 bytes 2026-02-20T22:33:18.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6210020352 bytes 2026-02-20T22:33:18.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6288238592 bytes 2026-02-20T22:33:18.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6321371136 bytes 2026-02-20T22:33:18.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6386075648 bytes 2026-02-20T22:33:18.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6439300096 bytes 2026-02-20T22:33:18.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6470934528 bytes 2026-02-20T22:33:18.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6538713088 bytes 2026-02-20T22:33:18.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6620773376 bytes 2026-02-20T22:33:18.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6688046080 bytes 2026-02-20T22:33:18.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6716013568 bytes 2026-02-20T22:33:18.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6755935232 bytes 2026-02-20T22:33:18.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6822430720 bytes 2026-02-20T22:33:18.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6831170560 bytes 2026-02-20T22:33:18.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6854065152 bytes 2026-02-20T22:33:18.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6868183040 bytes 2026-02-20T22:33:18.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 6894295040 bytes 2026-02-20T22:33:18.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 6902428672 bytes 2026-02-20T22:33:18.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 6962752512 bytes 2026-02-20T22:33:19.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7003196416 bytes 2026-02-20T22:33:19.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7046857728 bytes 2026-02-20T22:33:19.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7139017728 bytes 2026-02-20T22:33:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7139644416 bytes 2026-02-20T22:33:19.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7264602112 bytes 2026-02-20T22:33:19.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7281293312 bytes 2026-02-20T22:33:19.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7281899520 bytes 2026-02-20T22:33:19.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7311972352 bytes 2026-02-20T22:33:19.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7315942400 bytes 2026-02-20T22:33:19.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7322729472 bytes 2026-02-20T22:33:19.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7323344896 bytes 2026-02-20T22:33:19.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7356451840 bytes 2026-02-20T22:33:19.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7396570112 bytes 2026-02-20T22:33:19.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7428487168 bytes 2026-02-20T22:33:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7460367360 bytes 2026-02-20T22:33:19.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7488523264 bytes 2026-02-20T22:33:20.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7558134784 bytes 2026-02-20T22:33:20.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7595655168 bytes 2026-02-20T22:33:20.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7611621376 bytes 2026-02-20T22:33:20.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7647383552 bytes 2026-02-20T22:33:20.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7670020096 bytes 2026-02-20T22:33:20.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7755058176 bytes 2026-02-20T22:33:20.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 7795320832 bytes 2026-02-20T22:33:20.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 7853264896 bytes 2026-02-20T22:33:20.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 7905835008 bytes 2026-02-20T22:33:20.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 7967592448 bytes 2026-02-20T22:33:20.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 7990909952 bytes 2026-02-20T22:33:20.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 7991494656 bytes 2026-02-20T22:33:20.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8008803328 bytes 2026-02-20T22:33:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8083017728 bytes 2026-02-20T22:33:20.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8132313088 bytes 2026-02-20T22:33:20.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8177838080 bytes 2026-02-20T22:33:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8245472256 bytes 2026-02-20T22:33:20.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8295495680 bytes 2026-02-20T22:33:21.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8324200448 bytes 2026-02-20T22:33:21.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8374192128 bytes 2026-02-20T22:33:21.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8428618752 bytes 2026-02-20T22:33:21.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8446182400 bytes 2026-02-20T22:33:21.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8463476736 bytes 2026-02-20T22:33:21.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 8504188928 bytes 2026-02-20T22:33:21.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 8586257408 bytes 2026-02-20T22:33:21.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 8645774336 bytes 2026-02-20T22:33:21.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 8666791936 bytes 2026-02-20T22:33:21.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 8685383680 bytes 2026-02-20T22:33:21.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 8724280320 bytes 2026-02-20T22:33:21.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 8742206464 bytes 2026-02-20T22:33:21.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 8766660608 bytes 2026-02-20T22:33:21.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 8833267712 bytes 2026-02-20T22:33:21.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 8869657600 bytes 2026-02-20T22:33:21.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 8953487360 bytes 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:21.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:33:21.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:33:22.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:33:22.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:33:22.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"} 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:32:57 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:33:22.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:33:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:33:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:33:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:32:57' 2026-02-20T22:33:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:33:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:33:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:33:22.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"} 2026-02-20T22:33:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:33:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:33:22.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:33:22.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:33:22.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:33:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:33:22.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:33:22.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:32:57' 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:32:57' 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:32:57' 2026-02-20T22:33:22.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:33:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:33:22.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:33:22.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOK45104513mirror.1trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":535817216.0,"last_snapshot_bytes":535299072,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626729,"remote_snapshot_timestamp":1771626729,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:32:57cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:32:57' 2026-02-20T22:33:22.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:33:22.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:33:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:32:57' 2026-02-20T22:33:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:33:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:33:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:33:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:33:22.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap9 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap9 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9 2026-02-20T22:33:22.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:33:46.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9436391243191324) > weight (0.25) 2026-02-20T22:33:46.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.33466484236092187) > weight (0.25) 2026-02-20T22:33:46.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5397130614981507) > weight (0.25) 2026-02-20T22:33:46.030 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7121407730100812) > weight (0.25) 2026-02-20T22:33:46.030 INFO:tasks.rbd_mirror_thrash:waiting for 74.75706810436729 secs before thrashing 2026-02-20T22:33:51.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:33:51.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:33:51.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:33:51.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:33:52.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:33:52.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:33:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:33:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:33:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:33:52.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:33:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:33:52.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:33:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:33:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:33:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:33:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:33:52.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:33:52.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:33:52.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:33:52.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:33:52.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:33:52.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:33:52.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:33:52.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:33:52.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:33:52.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:33:52.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:52.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:33:53.679 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5765166757361965) > weight (0.25) 2026-02-20T22:33:53.680 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7949972656727867) > weight (0.25) 2026-02-20T22:33:53.680 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.46945253474695536) > weight (0.25) 2026-02-20T22:33:53.680 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8874036073077938) > weight (0.25) 2026-02-20T22:33:53.680 INFO:tasks.rbd_mirror_thrash:waiting for 75.14261465027984 secs before thrashing 2026-02-20T22:33:54.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 29 2026-02-20T22:33:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:33:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:33:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:33:54.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:33:54.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:33:54.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:33:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:33:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:54.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:54.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:33:54.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:33:54.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:54.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:33:54.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=29 2026-02-20T22:33:54.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:54.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:54.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:33:55.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:55.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:55.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:55.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:55.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 29 = '' 2026-02-20T22:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:33:55.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:55.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:55.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:55.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:55.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:55.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:55.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 29 = '' 2026-02-20T22:33:55.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:55.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:33:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:56.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:56.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:56.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:56.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:33:56.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:56.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:33:56.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:33:56.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:33:56.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:33:56.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 29 = '' 2026-02-20T22:33:56.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:33:56.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:33:58.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:33:58.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:33:58.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=29 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 29 = 29 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:33:58.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:33:59.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2026-02-20T22:33:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:33:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:33:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:33:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 18 2026-02-20T22:33:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:33:06 2026 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:33:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap8-clean ']' 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap8-clean 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8-clean 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:33:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:33:59.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:33:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:00.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:00.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:00.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:00.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap9 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:00.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:00.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:00.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:01.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:01.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:01.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:01.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:01.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap9-clean 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap9-clean 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9-clean 2026-02-20T22:34:01.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:34:02.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:34:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror test 2026-02-20T22:34:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:34:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:34:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:34:02.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2026-02-20T22:34:03.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 100777984 bytes 2026-02-20T22:34:03.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 119548928 bytes 2026-02-20T22:34:03.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 136763392 bytes 2026-02-20T22:34:03.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 199984128 bytes 2026-02-20T22:34:03.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 264882176 bytes 2026-02-20T22:34:03.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 301008896 bytes 2026-02-20T22:34:03.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 343284736 bytes 2026-02-20T22:34:03.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 436932608 bytes 2026-02-20T22:34:03.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 471484416 bytes 2026-02-20T22:34:03.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 476751872 bytes 2026-02-20T22:34:03.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 538163200 bytes 2026-02-20T22:34:04.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 560051200 bytes 2026-02-20T22:34:04.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 635826176 bytes 2026-02-20T22:34:04.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 636971008 bytes 2026-02-20T22:34:04.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 675774464 bytes 2026-02-20T22:34:04.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 693524480 bytes 2026-02-20T22:34:04.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 761716736 bytes 2026-02-20T22:34:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 803231744 bytes 2026-02-20T22:34:04.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 886431744 bytes 2026-02-20T22:34:04.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 930965504 bytes 2026-02-20T22:34:04.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 947328000 bytes 2026-02-20T22:34:04.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 992305152 bytes 2026-02-20T22:34:04.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1037365248 bytes 2026-02-20T22:34:04.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1087945728 bytes 2026-02-20T22:34:04.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1122362368 bytes 2026-02-20T22:34:04.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1145090048 bytes 2026-02-20T22:34:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1184458752 bytes 2026-02-20T22:34:04.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1211653120 bytes 2026-02-20T22:34:04.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1320184832 bytes 2026-02-20T22:34:05.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1340420096 bytes 2026-02-20T22:34:05.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1385753600 bytes 2026-02-20T22:34:05.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1386371072 bytes 2026-02-20T22:34:05.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1389374464 bytes 2026-02-20T22:34:05.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1440896000 bytes 2026-02-20T22:34:05.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1578096640 bytes 2026-02-20T22:34:05.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1620381696 bytes 2026-02-20T22:34:05.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1638712320 bytes 2026-02-20T22:34:05.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1664141312 bytes 2026-02-20T22:34:05.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1720482816 bytes 2026-02-20T22:34:05.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1750857728 bytes 2026-02-20T22:34:05.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1816212480 bytes 2026-02-20T22:34:05.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1827419136 bytes 2026-02-20T22:34:05.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1872561152 bytes 2026-02-20T22:34:05.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1957516288 bytes 2026-02-20T22:34:05.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1970081792 bytes 2026-02-20T22:34:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2003923968 bytes 2026-02-20T22:34:05.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2068443136 bytes 2026-02-20T22:34:06.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2141796352 bytes 2026-02-20T22:34:06.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2210460672 bytes 2026-02-20T22:34:06.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2230284288 bytes 2026-02-20T22:34:06.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2235465728 bytes 2026-02-20T22:34:06.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2250039296 bytes 2026-02-20T22:34:06.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2300312576 bytes 2026-02-20T22:34:06.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2318296064 bytes 2026-02-20T22:34:06.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2321806336 bytes 2026-02-20T22:34:06.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2371240960 bytes 2026-02-20T22:34:06.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2393472000 bytes 2026-02-20T22:34:06.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2453382144 bytes 2026-02-20T22:34:06.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2548707328 bytes 2026-02-20T22:34:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2560428032 bytes 2026-02-20T22:34:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2561777664 bytes 2026-02-20T22:34:06.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2573981696 bytes 2026-02-20T22:34:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2611725312 bytes 2026-02-20T22:34:06.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2657684480 bytes 2026-02-20T22:34:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2728637440 bytes 2026-02-20T22:34:06.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2797373440 bytes 2026-02-20T22:34:07.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2862768128 bytes 2026-02-20T22:34:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2897267712 bytes 2026-02-20T22:34:07.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2908563456 bytes 2026-02-20T22:34:07.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2942993408 bytes 2026-02-20T22:34:07.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2943603712 bytes 2026-02-20T22:34:07.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2962206720 bytes 2026-02-20T22:34:07.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2982722560 bytes 2026-02-20T22:34:07.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3001993216 bytes 2026-02-20T22:34:07.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3045552128 bytes 2026-02-20T22:34:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3062244352 bytes 2026-02-20T22:34:07.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3112117248 bytes 2026-02-20T22:34:07.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3112714240 bytes 2026-02-20T22:34:07.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3155897344 bytes 2026-02-20T22:34:07.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3183977472 bytes 2026-02-20T22:34:07.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3221995520 bytes 2026-02-20T22:34:07.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3264928768 bytes 2026-02-20T22:34:07.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3299336192 bytes 2026-02-20T22:34:07.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3322063872 bytes 2026-02-20T22:34:07.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3339805696 bytes 2026-02-20T22:34:07.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3404299264 bytes 2026-02-20T22:34:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3472468992 bytes 2026-02-20T22:34:08.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3575549952 bytes 2026-02-20T22:34:08.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3604640768 bytes 2026-02-20T22:34:08.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3700707328 bytes 2026-02-20T22:34:08.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3720906752 bytes 2026-02-20T22:34:08.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3807680512 bytes 2026-02-20T22:34:08.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3872693248 bytes 2026-02-20T22:34:08.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3883842560 bytes 2026-02-20T22:34:08.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3929475072 bytes 2026-02-20T22:34:08.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3953623040 bytes 2026-02-20T22:34:08.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4010135552 bytes 2026-02-20T22:34:08.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4010729472 bytes 2026-02-20T22:34:08.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4058781696 bytes 2026-02-20T22:34:08.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4095819776 bytes 2026-02-20T22:34:08.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4142446592 bytes 2026-02-20T22:34:08.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4173580288 bytes 2026-02-20T22:34:08.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4225891328 bytes 2026-02-20T22:34:08.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4286509056 bytes 2026-02-20T22:34:09.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4287128576 bytes 2026-02-20T22:34:09.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4337721344 bytes 2026-02-20T22:34:09.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4408091648 bytes 2026-02-20T22:34:09.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4457286656 bytes 2026-02-20T22:34:09.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4501746688 bytes 2026-02-20T22:34:09.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4572675072 bytes 2026-02-20T22:34:09.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4592031744 bytes 2026-02-20T22:34:09.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4614093824 bytes 2026-02-20T22:34:09.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4681106432 bytes 2026-02-20T22:34:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4746580992 bytes 2026-02-20T22:34:09.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4775232512 bytes 2026-02-20T22:34:09.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4831989760 bytes 2026-02-20T22:34:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4864075776 bytes 2026-02-20T22:34:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4864686080 bytes 2026-02-20T22:34:09.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4896940032 bytes 2026-02-20T22:34:09.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4917486592 bytes 2026-02-20T22:34:09.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4999775232 bytes 2026-02-20T22:34:10.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5031780352 bytes 2026-02-20T22:34:10.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5082490880 bytes 2026-02-20T22:34:10.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5169594368 bytes 2026-02-20T22:34:10.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5197804544 bytes 2026-02-20T22:34:10.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5252062208 bytes 2026-02-20T22:34:10.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5299046400 bytes 2026-02-20T22:34:10.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5328845824 bytes 2026-02-20T22:34:10.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5360264192 bytes 2026-02-20T22:34:10.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5436849152 bytes 2026-02-20T22:34:10.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5478922240 bytes 2026-02-20T22:34:10.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5523955712 bytes 2026-02-20T22:34:10.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5582731264 bytes 2026-02-20T22:34:10.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5695030272 bytes 2026-02-20T22:34:10.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5747664896 bytes 2026-02-20T22:34:10.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5758795776 bytes 2026-02-20T22:34:10.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5808464896 bytes 2026-02-20T22:34:11.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5894259712 bytes 2026-02-20T22:34:11.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5921762304 bytes 2026-02-20T22:34:11.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5988865024 bytes 2026-02-20T22:34:11.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6010184704 bytes 2026-02-20T22:34:11.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6097567744 bytes 2026-02-20T22:34:11.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6155074560 bytes 2026-02-20T22:34:11.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6180959232 bytes 2026-02-20T22:34:11.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6243613696 bytes 2026-02-20T22:34:11.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6283293696 bytes 2026-02-20T22:34:11.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6309019648 bytes 2026-02-20T22:34:11.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6364083200 bytes 2026-02-20T22:34:11.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6415117312 bytes 2026-02-20T22:34:11.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6477997056 bytes 2026-02-20T22:34:11.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6541230080 bytes 2026-02-20T22:34:11.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6600570880 bytes 2026-02-20T22:34:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6632638464 bytes 2026-02-20T22:34:11.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6678872064 bytes 2026-02-20T22:34:12.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6735897600 bytes 2026-02-20T22:34:12.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6786591744 bytes 2026-02-20T22:34:12.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6865153024 bytes 2026-02-20T22:34:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6892953600 bytes 2026-02-20T22:34:12.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6947551232 bytes 2026-02-20T22:34:12.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6969125888 bytes 2026-02-20T22:34:12.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6994491392 bytes 2026-02-20T22:34:12.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7047741440 bytes 2026-02-20T22:34:12.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7071279104 bytes 2026-02-20T22:34:12.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7101026304 bytes 2026-02-20T22:34:12.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7150081024 bytes 2026-02-20T22:34:12.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7196052480 bytes 2026-02-20T22:34:12.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7271468032 bytes 2026-02-20T22:34:12.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7278906368 bytes 2026-02-20T22:34:12.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7356737536 bytes 2026-02-20T22:34:12.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7391491072 bytes 2026-02-20T22:34:12.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7396486144 bytes 2026-02-20T22:34:12.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7441873920 bytes 2026-02-20T22:34:13.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7496361984 bytes 2026-02-20T22:34:13.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7529212928 bytes 2026-02-20T22:34:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7529840640 bytes 2026-02-20T22:34:13.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7570905088 bytes 2026-02-20T22:34:13.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7589200896 bytes 2026-02-20T22:34:13.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7589799936 bytes 2026-02-20T22:34:13.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7654780928 bytes 2026-02-20T22:34:13.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7667412992 bytes 2026-02-20T22:34:13.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7689080832 bytes 2026-02-20T22:34:13.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7737953280 bytes 2026-02-20T22:34:13.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7825360896 bytes 2026-02-20T22:34:13.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7833463808 bytes 2026-02-20T22:34:13.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7856932864 bytes 2026-02-20T22:34:13.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7905354752 bytes 2026-02-20T22:34:13.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7918281728 bytes 2026-02-20T22:34:13.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7949774848 bytes 2026-02-20T22:34:13.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7969991680 bytes 2026-02-20T22:34:13.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7979148288 bytes 2026-02-20T22:34:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8035306496 bytes 2026-02-20T22:34:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8092628992 bytes 2026-02-20T22:34:13.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8117756928 bytes 2026-02-20T22:34:14.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8142064640 bytes 2026-02-20T22:34:14.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8142684160 bytes 2026-02-20T22:34:14.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8174377984 bytes 2026-02-20T22:34:14.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8228665344 bytes 2026-02-20T22:34:14.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8284190720 bytes 2026-02-20T22:34:14.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8367409152 bytes 2026-02-20T22:34:14.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8384793600 bytes 2026-02-20T22:34:14.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8460191744 bytes 2026-02-20T22:34:14.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8493256704 bytes 2026-02-20T22:34:14.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8564687872 bytes 2026-02-20T22:34:14.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8631600128 bytes 2026-02-20T22:34:14.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8665789440 bytes 2026-02-20T22:34:14.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8685685760 bytes 2026-02-20T22:34:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8759035904 bytes 2026-02-20T22:34:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8808730624 bytes 2026-02-20T22:34:15.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8881946624 bytes 2026-02-20T22:34:15.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8969550848 bytes 2026-02-20T22:34:15.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8993472512 bytes 2026-02-20T22:34:15.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9029126144 bytes 2026-02-20T22:34:15.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9036119040 bytes 2026-02-20T22:34:15.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9091973120 bytes 2026-02-20T22:34:15.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9175764992 bytes 2026-02-20T22:34:15.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9223184384 bytes 2026-02-20T22:34:15.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9246671872 bytes 2026-02-20T22:34:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9266132992 bytes 2026-02-20T22:34:15.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9336880128 bytes 2026-02-20T22:34:15.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9394856960 bytes 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2026-02-20T22:34:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:34:15.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:34:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local description_pattern= 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local service_pattern= 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:34:16.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:34:16.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2026-02-20T22:34:16.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='test: 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: global_id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+replaying 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534658560.0,"last_snapshot_bytes":534018048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626780,"remote_snapshot_timestamp":1771626780,"replay_state":"idle"} 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: service: mirror.0 on trial107 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:33:57 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: peer_sites: 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: name: cluster2 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: state: up+stopped 2026-02-20T22:34:16.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: description: local image is primary 2026-02-20T22:34:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: last_update: 2026-02-20 22:33:57' 2026-02-20T22:34:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ state: .*up+replaying' 2026-02-20T22:34:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: state: up+replaying 2026-02-20T22:34:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^ description: .*' 2026-02-20T22:34:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534658560.0,"last_snapshot_bytes":534018048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626780,"remote_snapshot_timestamp":1771626780,"replay_state":"idle"} 2026-02-20T22:34:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:34:16.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:34:16.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep '^up+' 2026-02-20T22:34:16.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:34:16.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep 'service: *mirror..* on ' 2026-02-20T22:34:16.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: service: mirror.0 on trial107 2026-02-20T22:34:16.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update 2026-02-20T22:34:16.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:33:57' 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2026-02-20 22:33:57' 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state_pattern=up+replaying 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local 'prev_last_update=2026-02-20 22:33:57' 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ CEPH_ARGS= 2026-02-20T22:34:16.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2026-02-20T22:34:16.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status='OKOKOK142084263mirror.0trial10720.2.0-677-gf7870454trueOK42164271mirror.2trial10720.2.0-677-gf7870454falseOK43114314mirror.3trial10720.2.0-677-gf7870454falseOK45104513mirror.1trial10720.2.0-677-gf7870454falseOKtestd0cf40a8-e33d-4240-bedd-1a47bcf7db35up+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":534658560.0,"last_snapshot_bytes":534018048,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1771626780,"remote_snapshot_timestamp":1771626780,"replay_state":"idle"}42084263mirror.0trial1072026-02-20 22:33:57cluster2c3ececfc-6db9-428b-b666-3a05930e41b3up+stoppedlocal image is primary2026-02-20 22:33:57' 2026-02-20T22:34:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_update state 2026-02-20T22:34:16.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2026-02-20T22:34:16.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ last_update='2026-02-20 22:33:57' 2026-02-20T22:34:16.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2026-02-20T22:34:16.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=up+replaying 2026-02-20T22:34:16.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo up+replaying 2026-02-20T22:34:16.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep up+replaying 2026-02-20T22:34:16.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:up+replaying 2026-02-20T22:34:16.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:16.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap10 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap10 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10 2026-02-20T22:34:16.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:34:44.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:34:44.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:34:44.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:34:45.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:34:45.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:34:45.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:34:45.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:34:45.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:34:45.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:34:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:34:45.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:34:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:45.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:34:45.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:34:45.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:34:45.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:34:45.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:34:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:34:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:34:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:34:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:34:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:34:47.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 32 2026-02-20T22:34:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:34:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:34:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:34:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:34:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:34:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:34:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:34:47.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:34:47.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:34:47.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:34:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:47.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:34:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=32 2026-02-20T22:34:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:47.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:48.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:48.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:48.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:48.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:34:48.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:34:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:34:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 32 = '' 2026-02-20T22:34:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:48.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:34:48.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:48.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:48.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:48.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:48.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:48.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:48.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:48.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:34:48.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:34:48.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:34:48.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:34:48.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 32 = '' 2026-02-20T22:34:48.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:48.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:34:49.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:49.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:49.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:49.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:49.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:34:49.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:34:49.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:34:49.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:34:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 32 = '' 2026-02-20T22:34:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:49.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:34:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:51.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:51.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:51.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=32 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 32 = 32 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:51.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10 2026-02-20T22:34:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:34:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:34:51.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:34:52.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2026-02-20T22:34:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:34:52.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 20 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:33:06 2026 2026-02-20T22:34:52.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:34:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:34:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:34:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:34:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:34:52.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snap9-clean ']' 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap9-clean 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9-clean 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:52.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:52.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:52.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:53.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:53.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:53.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap10 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:53.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:53.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:34:53.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:53.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:54.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:34:54.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:54.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:34:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ clean_snap_name=snap10-clean 2026-02-20T22:34:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror test snap10-clean 2026-02-20T22:34:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10-clean 2026-02-20T22:34:54.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror test 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:34:55.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror test started 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:34:56.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:34:56.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:34:56.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:34:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:34:56.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:34:56.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:34:56.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:34:56.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:34:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:34:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:56.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:56.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:34:56.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2026-02-20T22:34:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:34:56.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:34:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:34:56.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:34:56.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror test 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:56.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2026-02-20T22:34:58.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 34 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror test 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-test.status 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-test.status 2026-02-20T22:34:58.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:34:58.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:58.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-test.status 2026-02-20T22:34:58.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-test.status 2026-02-20T22:34:58.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:34:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:58.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.status 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2026-02-20T22:34:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:58.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:34:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=34 2026-02-20T22:34:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:34:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:59.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:59.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:59.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:34:59.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:34:59.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:34:59.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:34:59.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 34 = '' 2026-02-20T22:34:59.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:59.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:34:59.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:34:59.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:34:59.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:34:59.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:34:59.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:34:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:34:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:34:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:34:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:34:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 34 = '' 2026-02-20T22:34:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:34:59.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:35:00.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:35:00.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:35:00.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:00.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:00.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.status 2026-02-20T22:35:00.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2026-02-20T22:35:00.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:35:00.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:35:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=34 2026-02-20T22:35:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 34 = 34 2026-02-20T22:35:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2026-02-20T22:35:00.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10-clean 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:35:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:35:00.788 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:35:00.788 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:35:00.789 INFO:tasks.rbd_mirror_thrash:waiting for 2.29427948262325 secs before reviving daemons 2026-02-20T22:35:00.789 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:35:00.781+0000 7f3773e2a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 25550) UID: 1000 2026-02-20T22:35:01.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'test': 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 21 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a724fdd651 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a724fdd651 2026-02-20T22:35:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:34:52 2026 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:26:29 2026 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: d0cf40a8-e33d-4240-bedd-1a47bcf7db35 2026-02-20T22:35:01.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:35:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 10 2026-02-20T22:35:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap1 2026-02-20T22:35:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap1 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:01.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:01.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:01.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:02.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:02.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:02.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap1-clean 2026-02-20T22:35:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap1-clean 2026-02-20T22:35:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:02.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap1-clean 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:02.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:03.084 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:35:03.084 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:35:03.085 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:35:03.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:35:03.085 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:35:03.087 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:35:03.087 INFO:tasks.rbd_mirror_thrash:waiting for 72.21683798831289 secs before thrashing 2026-02-20T22:35:03.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:03.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:03.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:03.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:04.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap2 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap2 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:04.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:04.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:04.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:04.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:05.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:05.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:05.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap2-clean 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap2-clean 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap2-clean 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:05.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:05.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:06.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:06.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:06.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:06.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap3 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap3 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:06.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:06.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:07.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:07.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:08.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:08.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap3-clean 2026-02-20T22:35:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap3-clean 2026-02-20T22:35:08.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap3-clean 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:08.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:08.824 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.856028433327918) > weight (0.25) 2026-02-20T22:35:08.824 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-20T22:35:08.825 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-20T22:35:08.825 INFO:tasks.rbd_mirror_thrash:waiting for 0.884434343544066 secs before reviving daemons 2026-02-20T22:35:08.826 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T22:35:08.816+0000 7fa35a622640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 25999) UID: 1000 2026-02-20T22:35:08.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:08.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:09.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:09.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:09.710 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-20T22:35:09.710 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:35:09.710 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-20T22:35:09.710 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-20T22:35:09.710 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-20T22:35:09.712 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-20T22:35:09.712 INFO:tasks.rbd_mirror_thrash:waiting for 82.74824088698573 secs before thrashing 2026-02-20T22:35:09.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap4 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap4 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:09.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:09.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:09.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:10.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:10.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:10.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:10.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:11.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:11.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:11.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap4-clean 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap4-clean 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap4-clean 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:11.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:11.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:11.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:12.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:12.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:12.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:12.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap5 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap5 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:12.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:13.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:13.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:14.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:14.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:14.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap5-clean 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap5-clean 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap5-clean 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:14.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:15.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:15.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:15.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap6 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap6 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:15.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:16.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:16.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:16.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:17.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:17.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:17.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap6-clean 2026-02-20T22:35:17.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap6-clean 2026-02-20T22:35:17.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap6-clean 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:17.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:17.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:18.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:18.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap7 2026-02-20T22:35:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap7 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:18.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:19.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:19.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:19.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:19.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap7-clean 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap7-clean 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:20.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap7-clean 2026-02-20T22:35:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:21.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:21.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:21.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:21.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:21.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap8 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap8 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:22.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:22.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:22.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:23.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:23.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:23.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:23.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap8-clean 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap8-clean 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap8-clean 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:23.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:23.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:23.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:24.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:24.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:24.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap9 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap9 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9 2026-02-20T22:35:24.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:24.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:24.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:24.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:24.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:25.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:25.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:25.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:25.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:26.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap9-clean 2026-02-20T22:35:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap9-clean 2026-02-20T22:35:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap9-clean 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:26.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:26.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:26.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:27.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:27.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap10 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap10 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:28.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:28.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:28.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:28.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap10-clean 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror test snap10-clean 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap10-clean 2026-02-20T22:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export 2026-02-20T22:35:29.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:29.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2026-02-20T22:35:30.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:30.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-test.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-test.export 2026-02-20T22:35:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:30.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 10 2026-02-20T22:35:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap1 2026-02-20T22:35:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap1 2026-02-20T22:35:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap1 2026-02-20T22:35:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2026-02-20T22:35:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:30.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap1-clean 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap1-clean 2026-02-20T22:35:30.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2026-02-20T22:35:31.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:31.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap2 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap2 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap2 2026-02-20T22:35:31.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2026-02-20T22:35:32.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap2-clean 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap2-clean 2026-02-20T22:35:32.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2026-02-20T22:35:33.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:33.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap3 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap3 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap3 2026-02-20T22:35:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2026-02-20T22:35:34.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap3-clean 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap3-clean 2026-02-20T22:35:34.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2026-02-20T22:35:35.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:35.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap4 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap4 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap4 2026-02-20T22:35:35.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2026-02-20T22:35:36.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap4-clean 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap4-clean 2026-02-20T22:35:36.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2026-02-20T22:35:37.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:37.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap5 2026-02-20T22:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap5 2026-02-20T22:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:37.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:37.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap5 2026-02-20T22:35:37.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2026-02-20T22:35:38.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap5-clean 2026-02-20T22:35:38.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2026-02-20T22:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap5-clean 2026-02-20T22:35:38.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2026-02-20T22:35:39.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:39.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap6 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap6 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap6 2026-02-20T22:35:39.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2026-02-20T22:35:40.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap6-clean 2026-02-20T22:35:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2026-02-20T22:35:40.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap6-clean 2026-02-20T22:35:40.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2026-02-20T22:35:41.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap7 2026-02-20T22:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap7 2026-02-20T22:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap7 2026-02-20T22:35:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2026-02-20T22:35:42.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap7-clean 2026-02-20T22:35:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2026-02-20T22:35:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:42.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:42.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:42.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap7-clean 2026-02-20T22:35:42.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2026-02-20T22:35:43.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap8 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap8 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:43.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap8 2026-02-20T22:35:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2026-02-20T22:35:44.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap8-clean 2026-02-20T22:35:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2026-02-20T22:35:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap8-clean 2026-02-20T22:35:44.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2026-02-20T22:35:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap9 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap9 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:45.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap9 2026-02-20T22:35:45.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2026-02-20T22:35:46.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap9-clean 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap9-clean 2026-02-20T22:35:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2026-02-20T22:35:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:47.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:35:47.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap10 2026-02-20T22:35:47.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap10 2026-02-20T22:35:47.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:47.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:47.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:47.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap10 2026-02-20T22:35:47.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2026-02-20T22:35:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap10-clean 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap10-clean 2026-02-20T22:35:48.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2026-02-20T22:35:49.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:35:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror test 2026-02-20T22:35:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:35:49.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:35:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror test 2026-02-20T22:35:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:49.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2026-02-20T22:35:49.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/test 2026-02-20T22:35:49.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:35:49.952+0000 7f991c6af640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5602c4be5040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:35:49.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:35:49.952+0000 7f99226bb640 -1 rbd::mirror::ImageReplayer: 0x5602c5547180 [3/d0cf40a8-e33d-4240-bedd-1a47bcf7db35] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:35:50.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror test deleted 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror test 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:35:50.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:35:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=test 2026-02-20T22:35:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-20T22:35:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:35:50.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-20T22:35:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:35:51.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:35:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:35:51.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror test deleted '' 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=test 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror test 2026-02-20T22:35:51.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:35:51.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:35:51.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=test 2026-02-20T22:35:51.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/test 2026-02-20T22:35:51.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:35:51.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image test: (2) No such file or directory 2026-02-20T22:35:51.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:35:51.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:35:51.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:35:51.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:51.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: create many images' 2026-02-20T22:35:51.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:35:51.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:35:51.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:35:51 TEST: create many images 2026-02-20T22:35:51.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:35:51 TEST: create many images 2026-02-20T22:35:51.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ snap_name=snap 2026-02-20T22:35:51.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 50 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_1 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_1 128M 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:35:51.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:35:51.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2026-02-20T22:35:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:35:51.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2026-02-20T22:35:51.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:35:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2026-02-20T22:35:51.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_1': 2026-02-20T22:35:51.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:35:51.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1265fc73da8a 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1265fc73da8a 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:35:51 2026 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:35:51 2026 2026-02-20T22:35:51.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:35:51 2026 2026-02-20T22:35:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:35:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:35:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3ec1d56b-1fff-4c3b-a39e-7c1c401a9ca7 2026-02-20T22:35:51.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_1 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:35:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:35:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:35:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:35:51.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:35:51.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 27293696 bytes 2026-02-20T22:35:52.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 66151424 bytes 2026-02-20T22:35:52.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 103047168 bytes 2026-02-20T22:35:52.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 126720000 bytes 2026-02-20T22:35:52.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 164130816 bytes 2026-02-20T22:35:52.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 215773184 bytes 2026-02-20T22:35:52.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 288359424 bytes 2026-02-20T22:35:52.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 325658624 bytes 2026-02-20T22:35:52.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 374247424 bytes 2026-02-20T22:35:52.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 394055680 bytes 2026-02-20T22:35:52.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 422994944 bytes 2026-02-20T22:35:52.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 451869696 bytes 2026-02-20T22:35:52.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 492696576 bytes 2026-02-20T22:35:52.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 517901312 bytes 2026-02-20T22:35:52.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 644008960 bytes 2026-02-20T22:35:52.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 683010048 bytes 2026-02-20T22:35:52.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 754911232 bytes 2026-02-20T22:35:52.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 798770176 bytes 2026-02-20T22:35:52.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 836580352 bytes 2026-02-20T22:35:52.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 854383616 bytes 2026-02-20T22:35:53.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 919599104 bytes 2026-02-20T22:35:53.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 953089024 bytes 2026-02-20T22:35:53.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 974173184 bytes 2026-02-20T22:35:53.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1011519488 bytes 2026-02-20T22:35:53.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1038145536 bytes 2026-02-20T22:35:53.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1105777664 bytes 2026-02-20T22:35:53.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1138121728 bytes 2026-02-20T22:35:53.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1211006976 bytes 2026-02-20T22:35:53.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1226593280 bytes 2026-02-20T22:35:53.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1239105536 bytes 2026-02-20T22:35:53.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1265725440 bytes 2026-02-20T22:35:53.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1293057024 bytes 2026-02-20T22:35:53.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1306000384 bytes 2026-02-20T22:35:53.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1332076544 bytes 2026-02-20T22:35:53.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1370271744 bytes 2026-02-20T22:35:53.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1408738304 bytes 2026-02-20T22:35:53.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1448190976 bytes 2026-02-20T22:35:53.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1449526272 bytes 2026-02-20T22:35:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1471927296 bytes 2026-02-20T22:35:53.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1525312512 bytes 2026-02-20T22:35:53.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1545822208 bytes 2026-02-20T22:35:53.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1580847104 bytes 2026-02-20T22:35:54.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1689811968 bytes 2026-02-20T22:35:54.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1697293312 bytes 2026-02-20T22:35:54.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1707595776 bytes 2026-02-20T22:35:54.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1786600448 bytes 2026-02-20T22:35:54.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1797902336 bytes 2026-02-20T22:35:54.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1838856192 bytes 2026-02-20T22:35:54.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1923875840 bytes 2026-02-20T22:35:54.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1933052928 bytes 2026-02-20T22:35:54.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1950035968 bytes 2026-02-20T22:35:54.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1993950208 bytes 2026-02-20T22:35:54.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2018490368 bytes 2026-02-20T22:35:54.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2036545536 bytes 2026-02-20T22:35:54.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2067872768 bytes 2026-02-20T22:35:54.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2071396352 bytes 2026-02-20T22:35:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2118209536 bytes 2026-02-20T22:35:54.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2136391680 bytes 2026-02-20T22:35:54.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2216823808 bytes 2026-02-20T22:35:54.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2328721408 bytes 2026-02-20T22:35:54.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2366109696 bytes 2026-02-20T22:35:55.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2485557248 bytes 2026-02-20T22:35:55.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2488498176 bytes 2026-02-20T22:35:55.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2517157888 bytes 2026-02-20T22:35:55.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2589983744 bytes 2026-02-20T22:35:55.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2603638784 bytes 2026-02-20T22:35:55.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2656706560 bytes 2026-02-20T22:35:55.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2729689088 bytes 2026-02-20T22:35:55.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2787936256 bytes 2026-02-20T22:35:55.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2788574208 bytes 2026-02-20T22:35:55.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2846340096 bytes 2026-02-20T22:35:55.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2917665792 bytes 2026-02-20T22:35:55.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2937934848 bytes 2026-02-20T22:35:55.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2979478528 bytes 2026-02-20T22:35:55.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3030756352 bytes 2026-02-20T22:35:55.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3100020736 bytes 2026-02-20T22:35:55.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3148766208 bytes 2026-02-20T22:35:55.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3210129408 bytes 2026-02-20T22:35:55.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3240493056 bytes 2026-02-20T22:35:56.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3288437760 bytes 2026-02-20T22:35:56.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3290029056 bytes 2026-02-20T22:35:56.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3344187392 bytes 2026-02-20T22:35:56.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3456091136 bytes 2026-02-20T22:35:56.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3483615232 bytes 2026-02-20T22:35:56.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3511568384 bytes 2026-02-20T22:35:56.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3537187840 bytes 2026-02-20T22:35:56.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3590460416 bytes 2026-02-20T22:35:56.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3595755520 bytes 2026-02-20T22:35:56.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3657633792 bytes 2026-02-20T22:35:56.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3709231104 bytes 2026-02-20T22:35:56.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3756855296 bytes 2026-02-20T22:35:56.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3770944512 bytes 2026-02-20T22:35:56.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3848638464 bytes 2026-02-20T22:35:56.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3887793152 bytes 2026-02-20T22:35:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3924822016 bytes 2026-02-20T22:35:56.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3954125824 bytes 2026-02-20T22:35:56.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4016368640 bytes 2026-02-20T22:35:57.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4059049984 bytes 2026-02-20T22:35:57.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4110271488 bytes 2026-02-20T22:35:57.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4137014272 bytes 2026-02-20T22:35:57.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4182318080 bytes 2026-02-20T22:35:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4218242048 bytes 2026-02-20T22:35:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4255938560 bytes 2026-02-20T22:35:57.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4312702976 bytes 2026-02-20T22:35:57.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4348337152 bytes 2026-02-20T22:35:57.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4441545728 bytes 2026-02-20T22:35:57.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4484199424 bytes 2026-02-20T22:35:57.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4502144000 bytes 2026-02-20T22:35:57.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4511121408 bytes 2026-02-20T22:35:57.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4573416448 bytes 2026-02-20T22:35:57.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4610430976 bytes 2026-02-20T22:35:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4632269824 bytes 2026-02-20T22:35:57.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4672538624 bytes 2026-02-20T22:35:57.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4738082816 bytes 2026-02-20T22:35:57.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4765372416 bytes 2026-02-20T22:35:58.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4824275968 bytes 2026-02-20T22:35:58.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4852907008 bytes 2026-02-20T22:35:58.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4892658688 bytes 2026-02-20T22:35:58.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4918793216 bytes 2026-02-20T22:35:58.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4926962688 bytes 2026-02-20T22:35:58.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4941721600 bytes 2026-02-20T22:35:58.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4988474368 bytes 2026-02-20T22:35:58.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5030370304 bytes 2026-02-20T22:35:58.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5116932096 bytes 2026-02-20T22:35:58.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5205387264 bytes 2026-02-20T22:35:58.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5226685440 bytes 2026-02-20T22:35:58.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5240066048 bytes 2026-02-20T22:35:58.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5303985152 bytes 2026-02-20T22:35:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5330017280 bytes 2026-02-20T22:35:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5408143360 bytes 2026-02-20T22:35:58.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5448888320 bytes 2026-02-20T22:35:58.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5487651840 bytes 2026-02-20T22:35:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5503574016 bytes 2026-02-20T22:35:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5546424320 bytes 2026-02-20T22:35:58.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5547385856 bytes 2026-02-20T22:35:59.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5581489152 bytes 2026-02-20T22:35:59.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5611821056 bytes 2026-02-20T22:35:59.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5663269888 bytes 2026-02-20T22:35:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5709337600 bytes 2026-02-20T22:35:59.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5793447936 bytes 2026-02-20T22:35:59.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5847781376 bytes 2026-02-20T22:35:59.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5902457856 bytes 2026-02-20T22:35:59.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5982931968 bytes 2026-02-20T22:35:59.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6055106560 bytes 2026-02-20T22:35:59.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6126948352 bytes 2026-02-20T22:35:59.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6145164288 bytes 2026-02-20T22:35:59.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6207931392 bytes 2026-02-20T22:35:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6268758016 bytes 2026-02-20T22:35:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6280356864 bytes 2026-02-20T22:35:59.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6296599552 bytes 2026-02-20T22:35:59.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6341563392 bytes 2026-02-20T22:35:59.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6399844352 bytes 2026-02-20T22:36:00.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6437468160 bytes 2026-02-20T22:36:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6494002176 bytes 2026-02-20T22:36:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6539689984 bytes 2026-02-20T22:36:00.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6647940096 bytes 2026-02-20T22:36:00.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6664274944 bytes 2026-02-20T22:36:00.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6699747328 bytes 2026-02-20T22:36:00.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6717009920 bytes 2026-02-20T22:36:00.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6755776512 bytes 2026-02-20T22:36:00.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6805925888 bytes 2026-02-20T22:36:00.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6869294080 bytes 2026-02-20T22:36:00.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6916150272 bytes 2026-02-20T22:36:00.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6980188160 bytes 2026-02-20T22:36:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7026186240 bytes 2026-02-20T22:36:00.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7052156928 bytes 2026-02-20T22:36:00.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7071572992 bytes 2026-02-20T22:36:00.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7133871104 bytes 2026-02-20T22:36:00.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7222278144 bytes 2026-02-20T22:36:01.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7310979072 bytes 2026-02-20T22:36:01.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7322391552 bytes 2026-02-20T22:36:01.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7350220800 bytes 2026-02-20T22:36:01.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7442125824 bytes 2026-02-20T22:36:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7456960512 bytes 2026-02-20T22:36:01.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7513487360 bytes 2026-02-20T22:36:01.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7621005312 bytes 2026-02-20T22:36:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7691613184 bytes 2026-02-20T22:36:01.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7781222400 bytes 2026-02-20T22:36:01.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7844368384 bytes 2026-02-20T22:36:01.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7922698240 bytes 2026-02-20T22:36:01.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7976750080 bytes 2026-02-20T22:36:01.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8060280832 bytes 2026-02-20T22:36:01.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8078129152 bytes 2026-02-20T22:36:01.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8123183104 bytes 2026-02-20T22:36:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8154070016 bytes 2026-02-20T22:36:02.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8196554752 bytes 2026-02-20T22:36:02.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8258605056 bytes 2026-02-20T22:36:02.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8267969536 bytes 2026-02-20T22:36:02.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8287677440 bytes 2026-02-20T22:36:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8332605440 bytes 2026-02-20T22:36:02.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8379873280 bytes 2026-02-20T22:36:02.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8387367936 bytes 2026-02-20T22:36:02.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8429928448 bytes 2026-02-20T22:36:02.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8436483072 bytes 2026-02-20T22:36:02.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8480003072 bytes 2026-02-20T22:36:02.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8534469632 bytes 2026-02-20T22:36:02.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8562519040 bytes 2026-02-20T22:36:02.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8592764928 bytes 2026-02-20T22:36:02.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8631493632 bytes 2026-02-20T22:36:02.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8661045248 bytes 2026-02-20T22:36:02.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8702066688 bytes 2026-02-20T22:36:02.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8778995712 bytes 2026-02-20T22:36:02.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8804320256 bytes 2026-02-20T22:36:02.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8869650432 bytes 2026-02-20T22:36:02.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8913244160 bytes 2026-02-20T22:36:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8913844224 bytes 2026-02-20T22:36:03.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9020283904 bytes 2026-02-20T22:36:03.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9046539264 bytes 2026-02-20T22:36:03.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9063761920 bytes 2026-02-20T22:36:03.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9115675648 bytes 2026-02-20T22:36:03.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9140597760 bytes 2026-02-20T22:36:03.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9187573760 bytes 2026-02-20T22:36:03.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9194148864 bytes 2026-02-20T22:36:03.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9244939264 bytes 2026-02-20T22:36:03.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9256142848 bytes 2026-02-20T22:36:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9272664064 bytes 2026-02-20T22:36:03.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9282385920 bytes 2026-02-20T22:36:03.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9308291072 bytes 2026-02-20T22:36:03.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9371021312 bytes 2026-02-20T22:36:03.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9386931200 bytes 2026-02-20T22:36:03.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9447328768 bytes 2026-02-20T22:36:03.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9523456000 bytes 2026-02-20T22:36:03.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9570594816 bytes 2026-02-20T22:36:03.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9591040000 bytes 2026-02-20T22:36:03.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9630618624 bytes 2026-02-20T22:36:04.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9690142720 bytes 2026-02-20T22:36:04.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 9779247104 bytes 2026-02-20T22:36:04.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 9850066944 bytes 2026-02-20T22:36:04.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 9890038784 bytes 2026-02-20T22:36:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 9920281600 bytes 2026-02-20T22:36:04.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 9962330112 bytes 2026-02-20T22:36:04.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23200 IOs, 10000486400 bytes 2026-02-20T22:36:04.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23300 IOs, 10074055680 bytes 2026-02-20T22:36:04.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23400 IOs, 10106971136 bytes 2026-02-20T22:36:04.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23500 IOs, 10123448320 bytes 2026-02-20T22:36:04.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23600 IOs, 10214643712 bytes 2026-02-20T22:36:04.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23700 IOs, 10238944256 bytes 2026-02-20T22:36:04.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23800 IOs, 10271952896 bytes 2026-02-20T22:36:04.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23900 IOs, 10299383808 bytes 2026-02-20T22:36:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:24000 IOs, 10355849216 bytes 2026-02-20T22:36:04.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_2 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_2 128M 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:36:04.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:04.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:04.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:04.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:04.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:04.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:36:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2026-02-20T22:36:05.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:05.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_2': 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 126eb74943b4 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.126eb74943b4 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:04 2026 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:04 2026 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:04 2026 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 241a16d7-c45c-4a2f-9678-2850dc15e21b 2026-02-20T22:36:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_2 2026-02-20T22:36:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:36:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:05.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:05.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 25274368 bytes 2026-02-20T22:36:06.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 62671872 bytes 2026-02-20T22:36:06.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 125854720 bytes 2026-02-20T22:36:06.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 126470144 bytes 2026-02-20T22:36:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 227086336 bytes 2026-02-20T22:36:06.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 291160064 bytes 2026-02-20T22:36:06.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 309856256 bytes 2026-02-20T22:36:06.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 364950528 bytes 2026-02-20T22:36:06.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 470189056 bytes 2026-02-20T22:36:06.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 474972160 bytes 2026-02-20T22:36:06.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 510812160 bytes 2026-02-20T22:36:06.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 550595584 bytes 2026-02-20T22:36:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 591707136 bytes 2026-02-20T22:36:06.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 631384064 bytes 2026-02-20T22:36:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 677361664 bytes 2026-02-20T22:36:06.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 693260288 bytes 2026-02-20T22:36:06.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 796057600 bytes 2026-02-20T22:36:06.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 851373056 bytes 2026-02-20T22:36:07.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 923596800 bytes 2026-02-20T22:36:07.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1019189248 bytes 2026-02-20T22:36:07.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1090857984 bytes 2026-02-20T22:36:07.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1137072128 bytes 2026-02-20T22:36:07.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1147293696 bytes 2026-02-20T22:36:07.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1197451264 bytes 2026-02-20T22:36:07.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1223995392 bytes 2026-02-20T22:36:07.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1257655296 bytes 2026-02-20T22:36:07.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1335849984 bytes 2026-02-20T22:36:07.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1371299840 bytes 2026-02-20T22:36:07.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1378984960 bytes 2026-02-20T22:36:07.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1454559232 bytes 2026-02-20T22:36:07.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1471340544 bytes 2026-02-20T22:36:07.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1524229120 bytes 2026-02-20T22:36:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1546734592 bytes 2026-02-20T22:36:07.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1627224064 bytes 2026-02-20T22:36:07.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1655487488 bytes 2026-02-20T22:36:07.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1669431296 bytes 2026-02-20T22:36:08.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1711901696 bytes 2026-02-20T22:36:08.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1755654144 bytes 2026-02-20T22:36:08.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1816155136 bytes 2026-02-20T22:36:08.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1849417728 bytes 2026-02-20T22:36:08.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1868810240 bytes 2026-02-20T22:36:08.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1913005056 bytes 2026-02-20T22:36:08.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1924992000 bytes 2026-02-20T22:36:08.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1977811968 bytes 2026-02-20T22:36:08.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2012516352 bytes 2026-02-20T22:36:08.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2048334848 bytes 2026-02-20T22:36:08.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2117665792 bytes 2026-02-20T22:36:08.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2144878592 bytes 2026-02-20T22:36:08.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2187695104 bytes 2026-02-20T22:36:08.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2218646528 bytes 2026-02-20T22:36:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2236006400 bytes 2026-02-20T22:36:08.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2371619840 bytes 2026-02-20T22:36:08.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2438460416 bytes 2026-02-20T22:36:08.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2441198592 bytes 2026-02-20T22:36:08.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2479547392 bytes 2026-02-20T22:36:09.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2546334720 bytes 2026-02-20T22:36:09.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2585147392 bytes 2026-02-20T22:36:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2684116992 bytes 2026-02-20T22:36:09.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2703524864 bytes 2026-02-20T22:36:09.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2778828800 bytes 2026-02-20T22:36:09.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2829308928 bytes 2026-02-20T22:36:09.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2872780800 bytes 2026-02-20T22:36:09.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2917278720 bytes 2026-02-20T22:36:09.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2946150400 bytes 2026-02-20T22:36:09.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3044483072 bytes 2026-02-20T22:36:09.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3088665600 bytes 2026-02-20T22:36:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3126261760 bytes 2026-02-20T22:36:09.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3142297600 bytes 2026-02-20T22:36:09.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3172876288 bytes 2026-02-20T22:36:09.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3199769600 bytes 2026-02-20T22:36:09.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3258932224 bytes 2026-02-20T22:36:10.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3346884608 bytes 2026-02-20T22:36:10.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3425689600 bytes 2026-02-20T22:36:10.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3454151680 bytes 2026-02-20T22:36:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3493929984 bytes 2026-02-20T22:36:10.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3552645120 bytes 2026-02-20T22:36:10.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3627545600 bytes 2026-02-20T22:36:10.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3656721408 bytes 2026-02-20T22:36:10.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3683277824 bytes 2026-02-20T22:36:10.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3708908544 bytes 2026-02-20T22:36:10.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3758767104 bytes 2026-02-20T22:36:10.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3794667520 bytes 2026-02-20T22:36:10.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3806184448 bytes 2026-02-20T22:36:10.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3870311424 bytes 2026-02-20T22:36:10.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3892878336 bytes 2026-02-20T22:36:10.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3923970048 bytes 2026-02-20T22:36:10.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3956867072 bytes 2026-02-20T22:36:10.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3969811456 bytes 2026-02-20T22:36:10.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4025484288 bytes 2026-02-20T22:36:11.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4110803968 bytes 2026-02-20T22:36:11.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4166734848 bytes 2026-02-20T22:36:11.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4232668160 bytes 2026-02-20T22:36:11.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4288201728 bytes 2026-02-20T22:36:11.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4330740736 bytes 2026-02-20T22:36:11.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4413264896 bytes 2026-02-20T22:36:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4474065920 bytes 2026-02-20T22:36:11.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4562887680 bytes 2026-02-20T22:36:11.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4599276544 bytes 2026-02-20T22:36:11.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4656551936 bytes 2026-02-20T22:36:11.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4692446208 bytes 2026-02-20T22:36:11.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4710809600 bytes 2026-02-20T22:36:11.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4755293184 bytes 2026-02-20T22:36:11.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4797201408 bytes 2026-02-20T22:36:11.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4832313344 bytes 2026-02-20T22:36:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4898963456 bytes 2026-02-20T22:36:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4906561536 bytes 2026-02-20T22:36:12.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4950083584 bytes 2026-02-20T22:36:12.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4975716352 bytes 2026-02-20T22:36:12.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 5014465536 bytes 2026-02-20T22:36:12.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5052400640 bytes 2026-02-20T22:36:12.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5123320832 bytes 2026-02-20T22:36:12.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5143517184 bytes 2026-02-20T22:36:12.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5245931520 bytes 2026-02-20T22:36:12.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5291271168 bytes 2026-02-20T22:36:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5328205824 bytes 2026-02-20T22:36:12.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5362069504 bytes 2026-02-20T22:36:12.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5415165952 bytes 2026-02-20T22:36:12.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5471266816 bytes 2026-02-20T22:36:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5512133632 bytes 2026-02-20T22:36:12.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5578191872 bytes 2026-02-20T22:36:12.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5611636736 bytes 2026-02-20T22:36:13.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5661407232 bytes 2026-02-20T22:36:13.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5709359104 bytes 2026-02-20T22:36:13.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5759548416 bytes 2026-02-20T22:36:13.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5801677824 bytes 2026-02-20T22:36:13.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5890460672 bytes 2026-02-20T22:36:13.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5892110336 bytes 2026-02-20T22:36:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5899876352 bytes 2026-02-20T22:36:13.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5939992576 bytes 2026-02-20T22:36:13.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5959415808 bytes 2026-02-20T22:36:13.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5982139392 bytes 2026-02-20T22:36:13.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6022205440 bytes 2026-02-20T22:36:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6085392384 bytes 2026-02-20T22:36:13.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6133257216 bytes 2026-02-20T22:36:13.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6203435008 bytes 2026-02-20T22:36:13.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6204055552 bytes 2026-02-20T22:36:13.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6205675520 bytes 2026-02-20T22:36:13.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6229512192 bytes 2026-02-20T22:36:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6234896384 bytes 2026-02-20T22:36:13.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6289826816 bytes 2026-02-20T22:36:14.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6314441728 bytes 2026-02-20T22:36:14.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6368352256 bytes 2026-02-20T22:36:14.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6461870080 bytes 2026-02-20T22:36:14.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6482138112 bytes 2026-02-20T22:36:14.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6492274688 bytes 2026-02-20T22:36:14.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6502560768 bytes 2026-02-20T22:36:14.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6568455168 bytes 2026-02-20T22:36:14.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6591890432 bytes 2026-02-20T22:36:14.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6592532480 bytes 2026-02-20T22:36:14.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6640382976 bytes 2026-02-20T22:36:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6663985152 bytes 2026-02-20T22:36:14.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6687129600 bytes 2026-02-20T22:36:14.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6692727808 bytes 2026-02-20T22:36:14.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6711221248 bytes 2026-02-20T22:36:14.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6718448640 bytes 2026-02-20T22:36:14.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6754642944 bytes 2026-02-20T22:36:14.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6777683968 bytes 2026-02-20T22:36:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6798113792 bytes 2026-02-20T22:36:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6853029888 bytes 2026-02-20T22:36:15.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6911119360 bytes 2026-02-20T22:36:15.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6960968704 bytes 2026-02-20T22:36:15.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7055064064 bytes 2026-02-20T22:36:15.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7115486208 bytes 2026-02-20T22:36:15.305 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9294391788132321) > weight (0.25) 2026-02-20T22:36:15.305 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9704961479364906) > weight (0.25) 2026-02-20T22:36:15.305 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7062763693172545) > weight (0.25) 2026-02-20T22:36:15.305 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.2715896748084242) > weight (0.25) 2026-02-20T22:36:15.305 INFO:tasks.rbd_mirror_thrash:waiting for 114.18182375628152 secs before thrashing 2026-02-20T22:36:15.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7164751872 bytes 2026-02-20T22:36:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7183130624 bytes 2026-02-20T22:36:15.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7223450624 bytes 2026-02-20T22:36:15.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7283615744 bytes 2026-02-20T22:36:15.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7316765696 bytes 2026-02-20T22:36:15.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7317362688 bytes 2026-02-20T22:36:15.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7333130240 bytes 2026-02-20T22:36:15.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7353108480 bytes 2026-02-20T22:36:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7375858688 bytes 2026-02-20T22:36:15.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7482130432 bytes 2026-02-20T22:36:15.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7506077696 bytes 2026-02-20T22:36:15.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7559593984 bytes 2026-02-20T22:36:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7582000128 bytes 2026-02-20T22:36:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7656929280 bytes 2026-02-20T22:36:16.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7690642432 bytes 2026-02-20T22:36:16.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7736546304 bytes 2026-02-20T22:36:16.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7784973312 bytes 2026-02-20T22:36:16.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7916057600 bytes 2026-02-20T22:36:16.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7983416320 bytes 2026-02-20T22:36:16.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8073581568 bytes 2026-02-20T22:36:16.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8107742208 bytes 2026-02-20T22:36:16.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8151484416 bytes 2026-02-20T22:36:16.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8214413312 bytes 2026-02-20T22:36:16.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8251357184 bytes 2026-02-20T22:36:16.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8290589696 bytes 2026-02-20T22:36:16.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8403219456 bytes 2026-02-20T22:36:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8417940480 bytes 2026-02-20T22:36:17.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8458296320 bytes 2026-02-20T22:36:17.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8581847040 bytes 2026-02-20T22:36:17.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8594072576 bytes 2026-02-20T22:36:17.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8608924672 bytes 2026-02-20T22:36:17.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8681226240 bytes 2026-02-20T22:36:17.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8739081216 bytes 2026-02-20T22:36:17.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8825272320 bytes 2026-02-20T22:36:17.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8845768704 bytes 2026-02-20T22:36:17.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8896100352 bytes 2026-02-20T22:36:17.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8978465792 bytes 2026-02-20T22:36:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8991327232 bytes 2026-02-20T22:36:17.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 9041829888 bytes 2026-02-20T22:36:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 9078266880 bytes 2026-02-20T22:36:17.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9097324544 bytes 2026-02-20T22:36:17.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9165258752 bytes 2026-02-20T22:36:18.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9218838528 bytes 2026-02-20T22:36:18.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9303072768 bytes 2026-02-20T22:36:18.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9344372736 bytes 2026-02-20T22:36:18.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9415931904 bytes 2026-02-20T22:36:18.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9438529536 bytes 2026-02-20T22:36:18.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9499204608 bytes 2026-02-20T22:36:18.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9542331392 bytes 2026-02-20T22:36:18.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9542916096 bytes 2026-02-20T22:36:18.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9552038912 bytes 2026-02-20T22:36:18.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9575198720 bytes 2026-02-20T22:36:18.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9682462720 bytes 2026-02-20T22:36:18.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9751309312 bytes 2026-02-20T22:36:18.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9785676800 bytes 2026-02-20T22:36:18.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9807047680 bytes 2026-02-20T22:36:18.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9842933760 bytes 2026-02-20T22:36:18.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9858715648 bytes 2026-02-20T22:36:18.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9883986944 bytes 2026-02-20T22:36:19.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9918069760 bytes 2026-02-20T22:36:19.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9957955584 bytes 2026-02-20T22:36:19.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 10029005824 bytes 2026-02-20T22:36:19.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 10049150976 bytes 2026-02-20T22:36:19.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 10101739520 bytes 2026-02-20T22:36:19.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 10149339136 bytes 2026-02-20T22:36:19.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 10155594752 bytes 2026-02-20T22:36:19.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 10263716864 bytes 2026-02-20T22:36:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 10267113472 bytes 2026-02-20T22:36:19.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23200 IOs, 10274650112 bytes 2026-02-20T22:36:19.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23300 IOs, 10286458880 bytes 2026-02-20T22:36:19.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23400 IOs, 10301008896 bytes 2026-02-20T22:36:19.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23500 IOs, 10373337088 bytes 2026-02-20T22:36:19.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23600 IOs, 10414105600 bytes 2026-02-20T22:36:19.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23700 IOs, 10425178112 bytes 2026-02-20T22:36:19.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23800 IOs, 10454947840 bytes 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:19.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_3 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_3 128M 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:19.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:19.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:36:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:19.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2026-02-20T22:36:20.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:20.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2026-02-20T22:36:20.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_3': 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1278416b695a 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1278416b695a 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:19 2026 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:19 2026 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:19 2026 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2e6164f9-d122-4af4-b228-4bf39a22c6fb 2026-02-20T22:36:20.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_3 2026-02-20T22:36:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:20.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:36:20.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=9 2026-02-20T22:36:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:20.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:21.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 19384320 bytes 2026-02-20T22:36:21.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 44601344 bytes 2026-02-20T22:36:21.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 86568960 bytes 2026-02-20T22:36:21.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 88769536 bytes 2026-02-20T22:36:21.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 179667968 bytes 2026-02-20T22:36:21.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 224598016 bytes 2026-02-20T22:36:21.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 242405376 bytes 2026-02-20T22:36:21.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 243015680 bytes 2026-02-20T22:36:21.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 309816320 bytes 2026-02-20T22:36:21.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 389701632 bytes 2026-02-20T22:36:21.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 446322688 bytes 2026-02-20T22:36:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 478833664 bytes 2026-02-20T22:36:21.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 520571904 bytes 2026-02-20T22:36:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 551689216 bytes 2026-02-20T22:36:21.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 611833856 bytes 2026-02-20T22:36:21.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 703098880 bytes 2026-02-20T22:36:21.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 743658496 bytes 2026-02-20T22:36:21.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 744278016 bytes 2026-02-20T22:36:22.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 744891392 bytes 2026-02-20T22:36:22.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 817654784 bytes 2026-02-20T22:36:22.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 851600384 bytes 2026-02-20T22:36:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 903693312 bytes 2026-02-20T22:36:22.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 936036352 bytes 2026-02-20T22:36:22.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 939169792 bytes 2026-02-20T22:36:22.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1026156544 bytes 2026-02-20T22:36:22.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1060499456 bytes 2026-02-20T22:36:22.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1103958016 bytes 2026-02-20T22:36:22.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1167314944 bytes 2026-02-20T22:36:22.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1203445760 bytes 2026-02-20T22:36:22.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1240295424 bytes 2026-02-20T22:36:22.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1324857344 bytes 2026-02-20T22:36:22.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1356937216 bytes 2026-02-20T22:36:22.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1379787776 bytes 2026-02-20T22:36:22.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1380402176 bytes 2026-02-20T22:36:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1391399936 bytes 2026-02-20T22:36:22.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1406800896 bytes 2026-02-20T22:36:23.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1407382528 bytes 2026-02-20T22:36:23.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1454048256 bytes 2026-02-20T22:36:23.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1496076288 bytes 2026-02-20T22:36:23.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1587196928 bytes 2026-02-20T22:36:23.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1593651200 bytes 2026-02-20T22:36:23.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1613093888 bytes 2026-02-20T22:36:23.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1613745152 bytes 2026-02-20T22:36:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1674017792 bytes 2026-02-20T22:36:23.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1699497984 bytes 2026-02-20T22:36:23.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1700532224 bytes 2026-02-20T22:36:23.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1701159936 bytes 2026-02-20T22:36:23.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1751733248 bytes 2026-02-20T22:36:23.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1830508544 bytes 2026-02-20T22:36:23.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1850237952 bytes 2026-02-20T22:36:23.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1869769728 bytes 2026-02-20T22:36:23.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1895014400 bytes 2026-02-20T22:36:23.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1912896512 bytes 2026-02-20T22:36:23.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1965981696 bytes 2026-02-20T22:36:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 1983398912 bytes 2026-02-20T22:36:24.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2039009280 bytes 2026-02-20T22:36:24.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2050827264 bytes 2026-02-20T22:36:24.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2096690176 bytes 2026-02-20T22:36:24.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2120456192 bytes 2026-02-20T22:36:24.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2121051136 bytes 2026-02-20T22:36:24.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2125065216 bytes 2026-02-20T22:36:24.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2213762048 bytes 2026-02-20T22:36:24.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2273932288 bytes 2026-02-20T22:36:24.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2331378688 bytes 2026-02-20T22:36:24.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2358916096 bytes 2026-02-20T22:36:24.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2403957760 bytes 2026-02-20T22:36:24.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2450856960 bytes 2026-02-20T22:36:24.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2463512576 bytes 2026-02-20T22:36:24.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2488308736 bytes 2026-02-20T22:36:24.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2502208512 bytes 2026-02-20T22:36:24.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2550343680 bytes 2026-02-20T22:36:24.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2550951936 bytes 2026-02-20T22:36:24.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2605127680 bytes 2026-02-20T22:36:24.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2667048960 bytes 2026-02-20T22:36:24.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2687454208 bytes 2026-02-20T22:36:25.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2707835904 bytes 2026-02-20T22:36:25.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2757129216 bytes 2026-02-20T22:36:25.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2811437056 bytes 2026-02-20T22:36:25.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2827919360 bytes 2026-02-20T22:36:25.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 2893360128 bytes 2026-02-20T22:36:25.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 2981488640 bytes 2026-02-20T22:36:25.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3031326720 bytes 2026-02-20T22:36:25.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3048365056 bytes 2026-02-20T22:36:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3106302976 bytes 2026-02-20T22:36:25.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3142634496 bytes 2026-02-20T22:36:25.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3183060992 bytes 2026-02-20T22:36:25.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3220979712 bytes 2026-02-20T22:36:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3244112896 bytes 2026-02-20T22:36:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3277381632 bytes 2026-02-20T22:36:25.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3278027776 bytes 2026-02-20T22:36:25.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3313034240 bytes 2026-02-20T22:36:25.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3359647744 bytes 2026-02-20T22:36:26.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3425743872 bytes 2026-02-20T22:36:26.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3451033600 bytes 2026-02-20T22:36:26.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3507593216 bytes 2026-02-20T22:36:26.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3592253440 bytes 2026-02-20T22:36:26.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3633598464 bytes 2026-02-20T22:36:26.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3655879680 bytes 2026-02-20T22:36:26.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3673199616 bytes 2026-02-20T22:36:26.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3705797632 bytes 2026-02-20T22:36:26.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3766257664 bytes 2026-02-20T22:36:26.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 3803023360 bytes 2026-02-20T22:36:26.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 3874085888 bytes 2026-02-20T22:36:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 3949868032 bytes 2026-02-20T22:36:26.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4000593920 bytes 2026-02-20T22:36:27.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4057108480 bytes 2026-02-20T22:36:27.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4107149312 bytes 2026-02-20T22:36:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4136586240 bytes 2026-02-20T22:36:27.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4211312640 bytes 2026-02-20T22:36:27.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4224541696 bytes 2026-02-20T22:36:27.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4242021376 bytes 2026-02-20T22:36:27.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4242633728 bytes 2026-02-20T22:36:27.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4294517760 bytes 2026-02-20T22:36:27.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4315957248 bytes 2026-02-20T22:36:27.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4425752576 bytes 2026-02-20T22:36:27.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4450788352 bytes 2026-02-20T22:36:27.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4470411264 bytes 2026-02-20T22:36:27.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4540548096 bytes 2026-02-20T22:36:27.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4611681280 bytes 2026-02-20T22:36:27.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4613922816 bytes 2026-02-20T22:36:27.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4665847808 bytes 2026-02-20T22:36:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4759462912 bytes 2026-02-20T22:36:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4823664640 bytes 2026-02-20T22:36:28.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4869798912 bytes 2026-02-20T22:36:28.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4957597696 bytes 2026-02-20T22:36:28.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5010743296 bytes 2026-02-20T22:36:28.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5155604480 bytes 2026-02-20T22:36:28.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5220458496 bytes 2026-02-20T22:36:28.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5297205248 bytes 2026-02-20T22:36:28.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5367364608 bytes 2026-02-20T22:36:28.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5416355840 bytes 2026-02-20T22:36:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5464128512 bytes 2026-02-20T22:36:28.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5474962432 bytes 2026-02-20T22:36:28.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5520402432 bytes 2026-02-20T22:36:28.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5584642048 bytes 2026-02-20T22:36:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5648323584 bytes 2026-02-20T22:36:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5670414336 bytes 2026-02-20T22:36:29.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5708081152 bytes 2026-02-20T22:36:29.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5717405696 bytes 2026-02-20T22:36:29.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5726740480 bytes 2026-02-20T22:36:29.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5759399936 bytes 2026-02-20T22:36:29.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5843635200 bytes 2026-02-20T22:36:29.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5896027136 bytes 2026-02-20T22:36:29.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5906868224 bytes 2026-02-20T22:36:29.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5964747776 bytes 2026-02-20T22:36:29.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5983731712 bytes 2026-02-20T22:36:29.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6025320448 bytes 2026-02-20T22:36:29.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6044809216 bytes 2026-02-20T22:36:29.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6115428352 bytes 2026-02-20T22:36:29.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6173557760 bytes 2026-02-20T22:36:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6229789696 bytes 2026-02-20T22:36:29.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_4 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:29.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_4 128M 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:36:29.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:29.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:29.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:29.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:29.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2026-02-20T22:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:29.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:29.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2026-02-20T22:36:30.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:30.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2026-02-20T22:36:31.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_4': 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12816df0650b 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12816df0650b 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:29 2026 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:29 2026 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:29 2026 2026-02-20T22:36:31.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ddee4f8b-4aff-4c02-80d5-06e133a89824 2026-02-20T22:36:31.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_4 2026-02-20T22:36:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:36:31.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=9 2026-02-20T22:36:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:31.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:31.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 42320896 bytes 2026-02-20T22:36:31.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 114788352 bytes 2026-02-20T22:36:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 191861760 bytes 2026-02-20T22:36:31.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 227660800 bytes 2026-02-20T22:36:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 278839296 bytes 2026-02-20T22:36:31.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 292827136 bytes 2026-02-20T22:36:31.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 303303680 bytes 2026-02-20T22:36:31.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 333925376 bytes 2026-02-20T22:36:31.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 390930432 bytes 2026-02-20T22:36:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 402840576 bytes 2026-02-20T22:36:31.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 417967104 bytes 2026-02-20T22:36:31.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 441661440 bytes 2026-02-20T22:36:31.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 450823168 bytes 2026-02-20T22:36:31.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 507791360 bytes 2026-02-20T22:36:31.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 514806784 bytes 2026-02-20T22:36:32.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 566707200 bytes 2026-02-20T22:36:32.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 578502656 bytes 2026-02-20T22:36:32.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 659501056 bytes 2026-02-20T22:36:32.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 699781120 bytes 2026-02-20T22:36:32.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 738942976 bytes 2026-02-20T22:36:32.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 807916544 bytes 2026-02-20T22:36:32.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 838729728 bytes 2026-02-20T22:36:32.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 864060416 bytes 2026-02-20T22:36:32.461 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-20T22:36:32.461 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-20T22:36:32.461 INFO:tasks.rbd_mirror_thrash:waiting for 5.23870303850749 secs before reviving daemons 2026-02-20T22:36:32.462 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:36:32.455+0000 7f74b8005640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 19527) UID: 1000 2026-02-20T22:36:32.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 900968448 bytes 2026-02-20T22:36:32.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 943645696 bytes 2026-02-20T22:36:32.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 944258048 bytes 2026-02-20T22:36:32.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 998414336 bytes 2026-02-20T22:36:32.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1073927168 bytes 2026-02-20T22:36:32.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1106326528 bytes 2026-02-20T22:36:32.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1158649856 bytes 2026-02-20T22:36:32.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1191173120 bytes 2026-02-20T22:36:32.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1254809600 bytes 2026-02-20T22:36:32.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1268916224 bytes 2026-02-20T22:36:33.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1299473408 bytes 2026-02-20T22:36:33.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1300097024 bytes 2026-02-20T22:36:33.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1360681984 bytes 2026-02-20T22:36:33.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1364836352 bytes 2026-02-20T22:36:33.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1395469312 bytes 2026-02-20T22:36:33.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1437753344 bytes 2026-02-20T22:36:33.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1488662528 bytes 2026-02-20T22:36:33.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1510253568 bytes 2026-02-20T22:36:33.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1530074112 bytes 2026-02-20T22:36:33.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1563866112 bytes 2026-02-20T22:36:33.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1615936512 bytes 2026-02-20T22:36:33.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1633379328 bytes 2026-02-20T22:36:33.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1652980736 bytes 2026-02-20T22:36:33.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1674052608 bytes 2026-02-20T22:36:33.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1763673088 bytes 2026-02-20T22:36:33.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1797661696 bytes 2026-02-20T22:36:33.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1832395776 bytes 2026-02-20T22:36:33.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1874138112 bytes 2026-02-20T22:36:33.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1930332160 bytes 2026-02-20T22:36:33.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1958380544 bytes 2026-02-20T22:36:34.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1999160320 bytes 2026-02-20T22:36:34.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2006303744 bytes 2026-02-20T22:36:34.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2044295168 bytes 2026-02-20T22:36:34.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2119896064 bytes 2026-02-20T22:36:34.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2175203328 bytes 2026-02-20T22:36:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2234768384 bytes 2026-02-20T22:36:34.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2276690944 bytes 2026-02-20T22:36:34.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2344721408 bytes 2026-02-20T22:36:34.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2349057024 bytes 2026-02-20T22:36:34.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2380611584 bytes 2026-02-20T22:36:34.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2404465664 bytes 2026-02-20T22:36:34.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2422816768 bytes 2026-02-20T22:36:34.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2468422656 bytes 2026-02-20T22:36:34.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2509774848 bytes 2026-02-20T22:36:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2588231680 bytes 2026-02-20T22:36:34.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2594361344 bytes 2026-02-20T22:36:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2618225664 bytes 2026-02-20T22:36:34.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2659792896 bytes 2026-02-20T22:36:34.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2706349056 bytes 2026-02-20T22:36:35.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2824097792 bytes 2026-02-20T22:36:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2865755136 bytes 2026-02-20T22:36:35.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2929924096 bytes 2026-02-20T22:36:35.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2975125504 bytes 2026-02-20T22:36:35.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3033208832 bytes 2026-02-20T22:36:35.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3068462080 bytes 2026-02-20T22:36:35.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3104709632 bytes 2026-02-20T22:36:35.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3137757184 bytes 2026-02-20T22:36:35.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3186565120 bytes 2026-02-20T22:36:35.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3231602688 bytes 2026-02-20T22:36:35.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3277965312 bytes 2026-02-20T22:36:35.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3309541376 bytes 2026-02-20T22:36:35.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3371870208 bytes 2026-02-20T22:36:35.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3429466112 bytes 2026-02-20T22:36:35.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3441790976 bytes 2026-02-20T22:36:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3481385984 bytes 2026-02-20T22:36:36.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3504350208 bytes 2026-02-20T22:36:36.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3552889856 bytes 2026-02-20T22:36:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3559525376 bytes 2026-02-20T22:36:36.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3613956096 bytes 2026-02-20T22:36:36.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3699818496 bytes 2026-02-20T22:36:36.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3713543168 bytes 2026-02-20T22:36:36.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3736878080 bytes 2026-02-20T22:36:36.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3804276736 bytes 2026-02-20T22:36:36.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3851531264 bytes 2026-02-20T22:36:36.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3870903296 bytes 2026-02-20T22:36:36.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3916884992 bytes 2026-02-20T22:36:36.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4000212992 bytes 2026-02-20T22:36:36.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4033126400 bytes 2026-02-20T22:36:36.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4047144960 bytes 2026-02-20T22:36:36.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4113063936 bytes 2026-02-20T22:36:36.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4175787008 bytes 2026-02-20T22:36:36.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4242265088 bytes 2026-02-20T22:36:36.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4246460416 bytes 2026-02-20T22:36:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4299550720 bytes 2026-02-20T22:36:37.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4334665728 bytes 2026-02-20T22:36:37.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4335276032 bytes 2026-02-20T22:36:37.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4369655808 bytes 2026-02-20T22:36:37.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4374067200 bytes 2026-02-20T22:36:37.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4385409024 bytes 2026-02-20T22:36:37.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4430124032 bytes 2026-02-20T22:36:37.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4462104576 bytes 2026-02-20T22:36:37.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4511162368 bytes 2026-02-20T22:36:37.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4538528768 bytes 2026-02-20T22:36:37.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4546300928 bytes 2026-02-20T22:36:37.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4556370944 bytes 2026-02-20T22:36:37.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4629873664 bytes 2026-02-20T22:36:37.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4653904896 bytes 2026-02-20T22:36:37.701 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-20T22:36:37.701 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:36:37.702 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-20T22:36:37.702 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-20T22:36:37.702 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-20T22:36:37.704 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-20T22:36:37.704 INFO:tasks.rbd_mirror_thrash:waiting for 66.31863268943773 secs before thrashing 2026-02-20T22:36:37.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4693971968 bytes 2026-02-20T22:36:37.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4731366400 bytes 2026-02-20T22:36:37.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4792428544 bytes 2026-02-20T22:36:37.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4915110912 bytes 2026-02-20T22:36:37.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4942942208 bytes 2026-02-20T22:36:38.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4985948160 bytes 2026-02-20T22:36:38.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5052571648 bytes 2026-02-20T22:36:38.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5097110528 bytes 2026-02-20T22:36:38.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5123078144 bytes 2026-02-20T22:36:38.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5143735296 bytes 2026-02-20T22:36:38.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5146316800 bytes 2026-02-20T22:36:38.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5205870592 bytes 2026-02-20T22:36:38.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5257840640 bytes 2026-02-20T22:36:38.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5288105984 bytes 2026-02-20T22:36:38.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5376596992 bytes 2026-02-20T22:36:38.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5422453760 bytes 2026-02-20T22:36:38.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5438924800 bytes 2026-02-20T22:36:38.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5511029760 bytes 2026-02-20T22:36:38.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5511643136 bytes 2026-02-20T22:36:38.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5570564096 bytes 2026-02-20T22:36:38.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5583499264 bytes 2026-02-20T22:36:38.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5603453952 bytes 2026-02-20T22:36:39.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5685902336 bytes 2026-02-20T22:36:39.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5712012288 bytes 2026-02-20T22:36:39.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5762621440 bytes 2026-02-20T22:36:39.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5815787520 bytes 2026-02-20T22:36:39.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5842807808 bytes 2026-02-20T22:36:39.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5873309696 bytes 2026-02-20T22:36:39.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 5896694784 bytes 2026-02-20T22:36:39.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 5936731136 bytes 2026-02-20T22:36:39.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 5991100416 bytes 2026-02-20T22:36:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6035727360 bytes 2026-02-20T22:36:39.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6086656000 bytes 2026-02-20T22:36:39.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6117909504 bytes 2026-02-20T22:36:39.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6129655808 bytes 2026-02-20T22:36:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6148367360 bytes 2026-02-20T22:36:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6214289408 bytes 2026-02-20T22:36:39.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6242443264 bytes 2026-02-20T22:36:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6248972288 bytes 2026-02-20T22:36:39.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6285785088 bytes 2026-02-20T22:36:39.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6325580800 bytes 2026-02-20T22:36:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6349537280 bytes 2026-02-20T22:36:40.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6412108800 bytes 2026-02-20T22:36:40.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6467352576 bytes 2026-02-20T22:36:40.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:40.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_5 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2026-02-20T22:36:40.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_5 128M 2026-02-20T22:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:40.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:36:40.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:40.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:40.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2026-02-20T22:36:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:40.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2026-02-20T22:36:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:36:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:40.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2026-02-20T22:36:40.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:40.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_5': 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 129155ac8d05 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.129155ac8d05 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:40.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:40 2026 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:40 2026 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:40 2026 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 6be28159-a310-434e-99b8-33aa6cef5760 2026-02-20T22:36:40.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:40.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_5 2026-02-20T22:36:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:36:40.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:36:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:40.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:40.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 61543424 bytes 2026-02-20T22:36:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 106625024 bytes 2026-02-20T22:36:40.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 184134656 bytes 2026-02-20T22:36:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 223989760 bytes 2026-02-20T22:36:40.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 224609280 bytes 2026-02-20T22:36:41.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 278499328 bytes 2026-02-20T22:36:41.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 298849280 bytes 2026-02-20T22:36:41.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 326943744 bytes 2026-02-20T22:36:41.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 373984256 bytes 2026-02-20T22:36:41.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 414362624 bytes 2026-02-20T22:36:41.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 524900352 bytes 2026-02-20T22:36:41.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 529492992 bytes 2026-02-20T22:36:41.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 544548864 bytes 2026-02-20T22:36:41.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 594333696 bytes 2026-02-20T22:36:41.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 670285824 bytes 2026-02-20T22:36:41.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 737597440 bytes 2026-02-20T22:36:41.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 759204864 bytes 2026-02-20T22:36:41.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 825101312 bytes 2026-02-20T22:36:41.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 870617088 bytes 2026-02-20T22:36:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 924725248 bytes 2026-02-20T22:36:41.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 938940416 bytes 2026-02-20T22:36:41.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 987489280 bytes 2026-02-20T22:36:41.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1033114624 bytes 2026-02-20T22:36:42.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1069072384 bytes 2026-02-20T22:36:42.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1101679616 bytes 2026-02-20T22:36:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1144848384 bytes 2026-02-20T22:36:42.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1188748288 bytes 2026-02-20T22:36:42.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1228869632 bytes 2026-02-20T22:36:42.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1294005248 bytes 2026-02-20T22:36:42.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1316515840 bytes 2026-02-20T22:36:42.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1429996544 bytes 2026-02-20T22:36:42.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1529590784 bytes 2026-02-20T22:36:42.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1553875968 bytes 2026-02-20T22:36:42.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1579636736 bytes 2026-02-20T22:36:42.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1602830336 bytes 2026-02-20T22:36:42.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1666412544 bytes 2026-02-20T22:36:42.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1667015680 bytes 2026-02-20T22:36:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1703175168 bytes 2026-02-20T22:36:42.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1764328448 bytes 2026-02-20T22:36:42.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1788959744 bytes 2026-02-20T22:36:43.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1845107712 bytes 2026-02-20T22:36:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1858289664 bytes 2026-02-20T22:36:43.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1880869888 bytes 2026-02-20T22:36:43.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1896378368 bytes 2026-02-20T22:36:43.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1928268800 bytes 2026-02-20T22:36:43.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1964194816 bytes 2026-02-20T22:36:43.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2015480832 bytes 2026-02-20T22:36:43.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2069161984 bytes 2026-02-20T22:36:43.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2093684736 bytes 2026-02-20T22:36:43.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2124721152 bytes 2026-02-20T22:36:43.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2133825536 bytes 2026-02-20T22:36:43.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2175534080 bytes 2026-02-20T22:36:43.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2203661312 bytes 2026-02-20T22:36:43.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2235813888 bytes 2026-02-20T22:36:43.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2261232640 bytes 2026-02-20T22:36:43.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2304864256 bytes 2026-02-20T22:36:43.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2357278720 bytes 2026-02-20T22:36:43.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2374508544 bytes 2026-02-20T22:36:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2416102400 bytes 2026-02-20T22:36:43.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2426143744 bytes 2026-02-20T22:36:44.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2434130944 bytes 2026-02-20T22:36:44.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2446330880 bytes 2026-02-20T22:36:44.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2542320640 bytes 2026-02-20T22:36:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2585091072 bytes 2026-02-20T22:36:44.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2597159936 bytes 2026-02-20T22:36:44.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2612180992 bytes 2026-02-20T22:36:44.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2662767616 bytes 2026-02-20T22:36:44.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2679038976 bytes 2026-02-20T22:36:44.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2742234112 bytes 2026-02-20T22:36:44.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2755043328 bytes 2026-02-20T22:36:44.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2802809856 bytes 2026-02-20T22:36:44.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2828667904 bytes 2026-02-20T22:36:44.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2911415296 bytes 2026-02-20T22:36:44.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2973691904 bytes 2026-02-20T22:36:44.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3009588224 bytes 2026-02-20T22:36:44.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3080338432 bytes 2026-02-20T22:36:44.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3117784064 bytes 2026-02-20T22:36:45.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3227474944 bytes 2026-02-20T22:36:45.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3270523904 bytes 2026-02-20T22:36:45.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3287913472 bytes 2026-02-20T22:36:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3310700544 bytes 2026-02-20T22:36:45.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3349429248 bytes 2026-02-20T22:36:45.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3386797056 bytes 2026-02-20T22:36:45.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3418377216 bytes 2026-02-20T22:36:45.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3420595200 bytes 2026-02-20T22:36:45.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3491077120 bytes 2026-02-20T22:36:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3515311104 bytes 2026-02-20T22:36:45.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3585630208 bytes 2026-02-20T22:36:45.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_6 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_6 128M 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:36:45.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:45.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2026-02-20T22:36:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:45.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:45.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2026-02-20T22:36:46.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:46.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2026-02-20T22:36:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_6': 2026-02-20T22:36:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 129a2f2a309c 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.129a2f2a309c 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:45 2026 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:45 2026 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:45 2026 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:46.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1b45a9ef-3d84-4767-b2e5-fe95a04ac4ac 2026-02-20T22:36:46.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_6 2026-02-20T22:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:36:46.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:46.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:46.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 44994560 bytes 2026-02-20T22:36:46.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 135572480 bytes 2026-02-20T22:36:46.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 184653824 bytes 2026-02-20T22:36:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 190681088 bytes 2026-02-20T22:36:47.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 267627520 bytes 2026-02-20T22:36:47.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 317566976 bytes 2026-02-20T22:36:47.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 349299712 bytes 2026-02-20T22:36:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 454247424 bytes 2026-02-20T22:36:47.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 482963456 bytes 2026-02-20T22:36:47.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 555054080 bytes 2026-02-20T22:36:47.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 561017856 bytes 2026-02-20T22:36:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 602421248 bytes 2026-02-20T22:36:47.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 626030592 bytes 2026-02-20T22:36:47.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 667292672 bytes 2026-02-20T22:36:47.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 692604928 bytes 2026-02-20T22:36:47.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 735951872 bytes 2026-02-20T22:36:47.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 767017984 bytes 2026-02-20T22:36:47.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 818910208 bytes 2026-02-20T22:36:47.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 835423232 bytes 2026-02-20T22:36:47.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 898930688 bytes 2026-02-20T22:36:47.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 942620672 bytes 2026-02-20T22:36:47.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 992443392 bytes 2026-02-20T22:36:48.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1019820032 bytes 2026-02-20T22:36:48.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1022472192 bytes 2026-02-20T22:36:48.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1077086208 bytes 2026-02-20T22:36:48.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1136379904 bytes 2026-02-20T22:36:48.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1202015232 bytes 2026-02-20T22:36:48.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1253751808 bytes 2026-02-20T22:36:48.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1292019712 bytes 2026-02-20T22:36:48.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1380037632 bytes 2026-02-20T22:36:48.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1472987136 bytes 2026-02-20T22:36:48.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1501777920 bytes 2026-02-20T22:36:48.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1542678528 bytes 2026-02-20T22:36:48.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1633659904 bytes 2026-02-20T22:36:48.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1658050560 bytes 2026-02-20T22:36:48.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1690213376 bytes 2026-02-20T22:36:48.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1738634240 bytes 2026-02-20T22:36:48.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1755004928 bytes 2026-02-20T22:36:48.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1803114496 bytes 2026-02-20T22:36:49.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1818109952 bytes 2026-02-20T22:36:49.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1852745728 bytes 2026-02-20T22:36:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1926458368 bytes 2026-02-20T22:36:49.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1955358720 bytes 2026-02-20T22:36:49.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1985315840 bytes 2026-02-20T22:36:49.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1985939456 bytes 2026-02-20T22:36:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2001994752 bytes 2026-02-20T22:36:49.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2082899968 bytes 2026-02-20T22:36:49.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2114569216 bytes 2026-02-20T22:36:49.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2137669632 bytes 2026-02-20T22:36:49.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2143787008 bytes 2026-02-20T22:36:49.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2144427008 bytes 2026-02-20T22:36:49.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2145029120 bytes 2026-02-20T22:36:49.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2188933120 bytes 2026-02-20T22:36:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2241743872 bytes 2026-02-20T22:36:49.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2272354304 bytes 2026-02-20T22:36:49.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2302423040 bytes 2026-02-20T22:36:49.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2363168768 bytes 2026-02-20T22:36:49.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2400181248 bytes 2026-02-20T22:36:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2410934272 bytes 2026-02-20T22:36:50.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2436813824 bytes 2026-02-20T22:36:50.086 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:36:50.079+0000 7f7ed1951640 -1 reset not still connected to 0x55c70c605a00 2026-02-20T22:36:50.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2503904256 bytes 2026-02-20T22:36:50.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2539523072 bytes 2026-02-20T22:36:50.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2579799040 bytes 2026-02-20T22:36:50.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2617316352 bytes 2026-02-20T22:36:50.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2692079616 bytes 2026-02-20T22:36:50.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2703109120 bytes 2026-02-20T22:36:50.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2708303872 bytes 2026-02-20T22:36:50.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2772823040 bytes 2026-02-20T22:36:50.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2778974208 bytes 2026-02-20T22:36:50.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2821327872 bytes 2026-02-20T22:36:50.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2852308992 bytes 2026-02-20T22:36:50.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2897139712 bytes 2026-02-20T22:36:50.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2915654656 bytes 2026-02-20T22:36:50.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2956390400 bytes 2026-02-20T22:36:50.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3004706816 bytes 2026-02-20T22:36:50.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3048206336 bytes 2026-02-20T22:36:50.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3064516608 bytes 2026-02-20T22:36:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3098892288 bytes 2026-02-20T22:36:51.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3161679872 bytes 2026-02-20T22:36:51.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3212328960 bytes 2026-02-20T22:36:51.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3258409984 bytes 2026-02-20T22:36:51.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3303332864 bytes 2026-02-20T22:36:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3329280000 bytes 2026-02-20T22:36:51.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3362717696 bytes 2026-02-20T22:36:51.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3445005312 bytes 2026-02-20T22:36:51.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3502676992 bytes 2026-02-20T22:36:51.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3555319808 bytes 2026-02-20T22:36:51.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3577843712 bytes 2026-02-20T22:36:51.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3607375872 bytes 2026-02-20T22:36:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3626911744 bytes 2026-02-20T22:36:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3664647168 bytes 2026-02-20T22:36:51.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3719301120 bytes 2026-02-20T22:36:51.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3751923712 bytes 2026-02-20T22:36:51.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3860346880 bytes 2026-02-20T22:36:52.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3939885056 bytes 2026-02-20T22:36:52.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4005561344 bytes 2026-02-20T22:36:52.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4074833920 bytes 2026-02-20T22:36:52.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4106918912 bytes 2026-02-20T22:36:52.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4119690240 bytes 2026-02-20T22:36:52.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4125825024 bytes 2026-02-20T22:36:52.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4160299008 bytes 2026-02-20T22:36:52.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4237646848 bytes 2026-02-20T22:36:52.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4261381120 bytes 2026-02-20T22:36:52.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4324006912 bytes 2026-02-20T22:36:52.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4379029504 bytes 2026-02-20T22:36:52.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4400386048 bytes 2026-02-20T22:36:52.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4421180416 bytes 2026-02-20T22:36:52.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4447842304 bytes 2026-02-20T22:36:52.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4509308928 bytes 2026-02-20T22:36:52.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4603495424 bytes 2026-02-20T22:36:52.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4627122176 bytes 2026-02-20T22:36:53.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4664004608 bytes 2026-02-20T22:36:53.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4702211072 bytes 2026-02-20T22:36:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4764954624 bytes 2026-02-20T22:36:53.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4787168256 bytes 2026-02-20T22:36:53.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4864130048 bytes 2026-02-20T22:36:53.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4883051520 bytes 2026-02-20T22:36:53.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4948454400 bytes 2026-02-20T22:36:53.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4994947072 bytes 2026-02-20T22:36:53.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5040040960 bytes 2026-02-20T22:36:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5075226624 bytes 2026-02-20T22:36:53.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5179407360 bytes 2026-02-20T22:36:53.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5252988928 bytes 2026-02-20T22:36:53.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5315483648 bytes 2026-02-20T22:36:53.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5417381888 bytes 2026-02-20T22:36:53.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5451182080 bytes 2026-02-20T22:36:54.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5550394368 bytes 2026-02-20T22:36:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5589541888 bytes 2026-02-20T22:36:54.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5629810688 bytes 2026-02-20T22:36:54.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5636683776 bytes 2026-02-20T22:36:54.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5663779840 bytes 2026-02-20T22:36:54.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5685431296 bytes 2026-02-20T22:36:54.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5716901888 bytes 2026-02-20T22:36:54.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5802506240 bytes 2026-02-20T22:36:54.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5827594240 bytes 2026-02-20T22:36:54.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5842990080 bytes 2026-02-20T22:36:54.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5863322624 bytes 2026-02-20T22:36:54.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5873586176 bytes 2026-02-20T22:36:54.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5894025216 bytes 2026-02-20T22:36:54.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5941105664 bytes 2026-02-20T22:36:54.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6000610304 bytes 2026-02-20T22:36:54.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6088144896 bytes 2026-02-20T22:36:54.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6160961536 bytes 2026-02-20T22:36:54.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6201047040 bytes 2026-02-20T22:36:55.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6238548992 bytes 2026-02-20T22:36:55.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6250752000 bytes 2026-02-20T22:36:55.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6336935936 bytes 2026-02-20T22:36:55.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6376961024 bytes 2026-02-20T22:36:55.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6429491200 bytes 2026-02-20T22:36:55.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6503010304 bytes 2026-02-20T22:36:55.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6529280000 bytes 2026-02-20T22:36:55.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6555652096 bytes 2026-02-20T22:36:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6582304768 bytes 2026-02-20T22:36:55.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6663847936 bytes 2026-02-20T22:36:55.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6696555520 bytes 2026-02-20T22:36:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6721758208 bytes 2026-02-20T22:36:55.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6785703936 bytes 2026-02-20T22:36:55.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6853956608 bytes 2026-02-20T22:36:55.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6898221056 bytes 2026-02-20T22:36:55.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6939673600 bytes 2026-02-20T22:36:56.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6954651648 bytes 2026-02-20T22:36:56.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6987879424 bytes 2026-02-20T22:36:56.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7037205504 bytes 2026-02-20T22:36:56.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7068651520 bytes 2026-02-20T22:36:56.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7154936832 bytes 2026-02-20T22:36:56.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7197768704 bytes 2026-02-20T22:36:56.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7236695040 bytes 2026-02-20T22:36:56.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7327081472 bytes 2026-02-20T22:36:56.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7365760000 bytes 2026-02-20T22:36:56.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7416248320 bytes 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_7 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2026-02-20T22:36:56.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_7 128M 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:56.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:36:56.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:36:56.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:36:56.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2026-02-20T22:36:57.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:36:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2026-02-20T22:36:57.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_7': 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12a4515da8c4 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12a4515da8c4 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:36:57.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:56 2026 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:56 2026 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:56 2026 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9795c60e-64e0-4de7-a6fb-f8429e83e276 2026-02-20T22:36:57.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_7 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:36:57.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:36:57.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:36:57.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 16660480 bytes 2026-02-20T22:36:57.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 57821184 bytes 2026-02-20T22:36:57.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 138295296 bytes 2026-02-20T22:36:57.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 163858432 bytes 2026-02-20T22:36:57.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 183109632 bytes 2026-02-20T22:36:58.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 240288768 bytes 2026-02-20T22:36:58.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 242893824 bytes 2026-02-20T22:36:58.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 318007296 bytes 2026-02-20T22:36:58.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 381231104 bytes 2026-02-20T22:36:58.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 422952960 bytes 2026-02-20T22:36:58.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 445517824 bytes 2026-02-20T22:36:58.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 458871808 bytes 2026-02-20T22:36:58.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 546374656 bytes 2026-02-20T22:36:58.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 563930112 bytes 2026-02-20T22:36:58.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 652010496 bytes 2026-02-20T22:36:58.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 699747328 bytes 2026-02-20T22:36:58.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 730277888 bytes 2026-02-20T22:36:58.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 751631360 bytes 2026-02-20T22:36:58.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 777733120 bytes 2026-02-20T22:36:58.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 778348544 bytes 2026-02-20T22:36:58.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 803393536 bytes 2026-02-20T22:36:58.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 812910592 bytes 2026-02-20T22:36:58.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 840997888 bytes 2026-02-20T22:36:59.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 869981184 bytes 2026-02-20T22:36:59.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 877801472 bytes 2026-02-20T22:36:59.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 921202688 bytes 2026-02-20T22:36:59.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 978392064 bytes 2026-02-20T22:36:59.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 993058816 bytes 2026-02-20T22:36:59.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1014129664 bytes 2026-02-20T22:36:59.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1054181376 bytes 2026-02-20T22:36:59.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1135168512 bytes 2026-02-20T22:36:59.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1205564416 bytes 2026-02-20T22:36:59.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1287271424 bytes 2026-02-20T22:36:59.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1339224064 bytes 2026-02-20T22:36:59.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1367787520 bytes 2026-02-20T22:36:59.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1379574784 bytes 2026-02-20T22:36:59.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1438601216 bytes 2026-02-20T22:36:59.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1478123520 bytes 2026-02-20T22:36:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1506251776 bytes 2026-02-20T22:36:59.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1562562560 bytes 2026-02-20T22:37:00.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1641900032 bytes 2026-02-20T22:37:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1681738752 bytes 2026-02-20T22:37:00.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1717477376 bytes 2026-02-20T22:37:00.133 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:37:00.127+0000 7f72ea4a6640 -1 reset not still connected to 0x55c8133ab5f0 2026-02-20T22:37:00.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1787200512 bytes 2026-02-20T22:37:00.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1805934592 bytes 2026-02-20T22:37:00.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1867634688 bytes 2026-02-20T22:37:00.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1873242112 bytes 2026-02-20T22:37:00.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1948411904 bytes 2026-02-20T22:37:00.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1970337792 bytes 2026-02-20T22:37:00.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2054632448 bytes 2026-02-20T22:37:00.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2072912896 bytes 2026-02-20T22:37:00.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2085447680 bytes 2026-02-20T22:37:00.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2094773248 bytes 2026-02-20T22:37:00.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2095385600 bytes 2026-02-20T22:37:00.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2149487616 bytes 2026-02-20T22:37:00.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2150787072 bytes 2026-02-20T22:37:00.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2219209728 bytes 2026-02-20T22:37:00.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2300363776 bytes 2026-02-20T22:37:01.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2342900736 bytes 2026-02-20T22:37:01.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2410060800 bytes 2026-02-20T22:37:01.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2462909440 bytes 2026-02-20T22:37:01.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2463511552 bytes 2026-02-20T22:37:01.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2490840064 bytes 2026-02-20T22:37:01.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2528439296 bytes 2026-02-20T22:37:01.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2616816640 bytes 2026-02-20T22:37:01.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2642023424 bytes 2026-02-20T22:37:01.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2709417984 bytes 2026-02-20T22:37:01.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2764577792 bytes 2026-02-20T22:37:01.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2813783040 bytes 2026-02-20T22:37:01.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2863314944 bytes 2026-02-20T22:37:01.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2942337024 bytes 2026-02-20T22:37:01.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2971982848 bytes 2026-02-20T22:37:01.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2999696384 bytes 2026-02-20T22:37:01.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3042164736 bytes 2026-02-20T22:37:02.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3047152640 bytes 2026-02-20T22:37:02.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3083154432 bytes 2026-02-20T22:37:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3145178112 bytes 2026-02-20T22:37:02.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3190238208 bytes 2026-02-20T22:37:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3263527936 bytes 2026-02-20T22:37:02.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3284771840 bytes 2026-02-20T22:37:02.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3326511104 bytes 2026-02-20T22:37:02.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3341005824 bytes 2026-02-20T22:37:02.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3347132416 bytes 2026-02-20T22:37:02.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3385500672 bytes 2026-02-20T22:37:02.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3422777344 bytes 2026-02-20T22:37:02.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3482841088 bytes 2026-02-20T22:37:02.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3552435200 bytes 2026-02-20T22:37:02.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3591914496 bytes 2026-02-20T22:37:02.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3676196864 bytes 2026-02-20T22:37:02.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3744761856 bytes 2026-02-20T22:37:02.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3745824768 bytes 2026-02-20T22:37:03.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3785417728 bytes 2026-02-20T22:37:03.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3847063552 bytes 2026-02-20T22:37:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3884525568 bytes 2026-02-20T22:37:03.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3959314432 bytes 2026-02-20T22:37:03.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4020626432 bytes 2026-02-20T22:37:03.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4064251904 bytes 2026-02-20T22:37:03.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4101467136 bytes 2026-02-20T22:37:03.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4123810816 bytes 2026-02-20T22:37:03.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4124406784 bytes 2026-02-20T22:37:03.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4154463232 bytes 2026-02-20T22:37:03.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4173046784 bytes 2026-02-20T22:37:03.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4173658112 bytes 2026-02-20T22:37:03.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:03.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_8 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_8 128M 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:03.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:03.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2026-02-20T22:37:04.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:04.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_8': 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12afeef70a04 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12afeef70a04 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:03 2026 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:03 2026 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:03 2026 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 25965088-a176-4775-96aa-3e7b420cb588 2026-02-20T22:37:04.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_8 2026-02-20T22:37:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:04.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:37:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=12 2026-02-20T22:37:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:04.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:04.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 64555008 bytes 2026-02-20T22:37:04.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 132230144 bytes 2026-02-20T22:37:04.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 132841472 bytes 2026-02-20T22:37:04.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 157589504 bytes 2026-02-20T22:37:04.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 208241664 bytes 2026-02-20T22:37:05.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 257703936 bytes 2026-02-20T22:37:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 291727360 bytes 2026-02-20T22:37:05.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 319343616 bytes 2026-02-20T22:37:05.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 359965696 bytes 2026-02-20T22:37:05.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 389549056 bytes 2026-02-20T22:37:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 416367616 bytes 2026-02-20T22:37:05.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 516748288 bytes 2026-02-20T22:37:05.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 584136704 bytes 2026-02-20T22:37:05.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 616151040 bytes 2026-02-20T22:37:05.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 647452672 bytes 2026-02-20T22:37:05.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 658590720 bytes 2026-02-20T22:37:05.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 680193024 bytes 2026-02-20T22:37:05.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 716655616 bytes 2026-02-20T22:37:05.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 756425728 bytes 2026-02-20T22:37:05.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 775622656 bytes 2026-02-20T22:37:05.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 816802816 bytes 2026-02-20T22:37:05.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 871596032 bytes 2026-02-20T22:37:05.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 900952064 bytes 2026-02-20T22:37:06.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 968109056 bytes 2026-02-20T22:37:06.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1024039936 bytes 2026-02-20T22:37:06.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1054752768 bytes 2026-02-20T22:37:06.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1117579264 bytes 2026-02-20T22:37:06.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1165965312 bytes 2026-02-20T22:37:06.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1240624128 bytes 2026-02-20T22:37:06.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1302301696 bytes 2026-02-20T22:37:06.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1377174528 bytes 2026-02-20T22:37:06.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1392449536 bytes 2026-02-20T22:37:06.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1459461120 bytes 2026-02-20T22:37:06.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1521368064 bytes 2026-02-20T22:37:06.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1535662080 bytes 2026-02-20T22:37:06.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1563522048 bytes 2026-02-20T22:37:06.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1600336896 bytes 2026-02-20T22:37:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1620484096 bytes 2026-02-20T22:37:06.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1651456000 bytes 2026-02-20T22:37:06.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1684644864 bytes 2026-02-20T22:37:07.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1733642240 bytes 2026-02-20T22:37:07.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1753816064 bytes 2026-02-20T22:37:07.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1813033984 bytes 2026-02-20T22:37:07.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1834569728 bytes 2026-02-20T22:37:07.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1864280064 bytes 2026-02-20T22:37:07.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1898588160 bytes 2026-02-20T22:37:07.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1953203200 bytes 2026-02-20T22:37:07.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2014834688 bytes 2026-02-20T22:37:07.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2077390848 bytes 2026-02-20T22:37:07.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2155949056 bytes 2026-02-20T22:37:07.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2289669120 bytes 2026-02-20T22:37:07.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2334281728 bytes 2026-02-20T22:37:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2334899200 bytes 2026-02-20T22:37:07.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2390763520 bytes 2026-02-20T22:37:07.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2391379968 bytes 2026-02-20T22:37:07.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2431618048 bytes 2026-02-20T22:37:07.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2473842688 bytes 2026-02-20T22:37:08.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2498541568 bytes 2026-02-20T22:37:08.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2536500224 bytes 2026-02-20T22:37:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2558295040 bytes 2026-02-20T22:37:08.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2627454976 bytes 2026-02-20T22:37:08.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2671949824 bytes 2026-02-20T22:37:08.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2709471232 bytes 2026-02-20T22:37:08.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2754089984 bytes 2026-02-20T22:37:08.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2840910848 bytes 2026-02-20T22:37:08.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2907122688 bytes 2026-02-20T22:37:08.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2923447296 bytes 2026-02-20T22:37:08.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2968202240 bytes 2026-02-20T22:37:08.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3041972224 bytes 2026-02-20T22:37:08.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3086357504 bytes 2026-02-20T22:37:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3119039488 bytes 2026-02-20T22:37:08.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3138232320 bytes 2026-02-20T22:37:08.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3165614080 bytes 2026-02-20T22:37:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3172477952 bytes 2026-02-20T22:37:09.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3226116096 bytes 2026-02-20T22:37:09.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3258271744 bytes 2026-02-20T22:37:09.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3330028544 bytes 2026-02-20T22:37:09.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3399209984 bytes 2026-02-20T22:37:09.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3421348864 bytes 2026-02-20T22:37:09.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3480551424 bytes 2026-02-20T22:37:09.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3500176384 bytes 2026-02-20T22:37:09.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3588708352 bytes 2026-02-20T22:37:09.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3653606400 bytes 2026-02-20T22:37:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3699749888 bytes 2026-02-20T22:37:09.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3727944704 bytes 2026-02-20T22:37:09.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3883649024 bytes 2026-02-20T22:37:09.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3884315648 bytes 2026-02-20T22:37:10.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3941049344 bytes 2026-02-20T22:37:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3979132928 bytes 2026-02-20T22:37:10.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4073838592 bytes 2026-02-20T22:37:10.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4097950720 bytes 2026-02-20T22:37:10.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4099609600 bytes 2026-02-20T22:37:10.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4160501760 bytes 2026-02-20T22:37:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4214530048 bytes 2026-02-20T22:37:10.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4263970816 bytes 2026-02-20T22:37:10.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4269576192 bytes 2026-02-20T22:37:10.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4290542592 bytes 2026-02-20T22:37:10.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4376368128 bytes 2026-02-20T22:37:10.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4469204992 bytes 2026-02-20T22:37:10.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4473657344 bytes 2026-02-20T22:37:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4504142848 bytes 2026-02-20T22:37:10.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4518099968 bytes 2026-02-20T22:37:10.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4533289984 bytes 2026-02-20T22:37:10.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4536762368 bytes 2026-02-20T22:37:10.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4565279744 bytes 2026-02-20T22:37:11.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4589170688 bytes 2026-02-20T22:37:11.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4662523904 bytes 2026-02-20T22:37:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4709808128 bytes 2026-02-20T22:37:11.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4710461440 bytes 2026-02-20T22:37:11.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4756533248 bytes 2026-02-20T22:37:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4833527808 bytes 2026-02-20T22:37:11.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4841167872 bytes 2026-02-20T22:37:11.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4853508096 bytes 2026-02-20T22:37:11.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4865402880 bytes 2026-02-20T22:37:11.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4967583744 bytes 2026-02-20T22:37:11.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4983296000 bytes 2026-02-20T22:37:11.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5054081024 bytes 2026-02-20T22:37:11.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5099010048 bytes 2026-02-20T22:37:11.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5099611136 bytes 2026-02-20T22:37:11.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5178179584 bytes 2026-02-20T22:37:11.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5221829632 bytes 2026-02-20T22:37:11.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5245724672 bytes 2026-02-20T22:37:11.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5260414976 bytes 2026-02-20T22:37:12.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5332225024 bytes 2026-02-20T22:37:12.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5345067008 bytes 2026-02-20T22:37:12.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5378413568 bytes 2026-02-20T22:37:12.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5424889856 bytes 2026-02-20T22:37:12.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5439024128 bytes 2026-02-20T22:37:12.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5449318400 bytes 2026-02-20T22:37:12.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5470251008 bytes 2026-02-20T22:37:12.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5489442816 bytes 2026-02-20T22:37:12.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5532277760 bytes 2026-02-20T22:37:12.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5556456448 bytes 2026-02-20T22:37:12.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5597638656 bytes 2026-02-20T22:37:12.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5646940160 bytes 2026-02-20T22:37:12.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5689491456 bytes 2026-02-20T22:37:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5704997888 bytes 2026-02-20T22:37:12.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5724732416 bytes 2026-02-20T22:37:12.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5749906432 bytes 2026-02-20T22:37:12.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5810317312 bytes 2026-02-20T22:37:12.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5835160576 bytes 2026-02-20T22:37:12.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5852168192 bytes 2026-02-20T22:37:12.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5894406144 bytes 2026-02-20T22:37:13.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5928423424 bytes 2026-02-20T22:37:13.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5980907520 bytes 2026-02-20T22:37:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6019832832 bytes 2026-02-20T22:37:13.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6040865792 bytes 2026-02-20T22:37:13.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6072158208 bytes 2026-02-20T22:37:13.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6148410368 bytes 2026-02-20T22:37:13.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6164115456 bytes 2026-02-20T22:37:13.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6216507392 bytes 2026-02-20T22:37:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6238273536 bytes 2026-02-20T22:37:13.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6248193024 bytes 2026-02-20T22:37:13.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6276972544 bytes 2026-02-20T22:37:13.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6300261376 bytes 2026-02-20T22:37:13.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6335650816 bytes 2026-02-20T22:37:13.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6375461888 bytes 2026-02-20T22:37:13.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6436884480 bytes 2026-02-20T22:37:13.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6480462848 bytes 2026-02-20T22:37:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6505375744 bytes 2026-02-20T22:37:13.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6572876800 bytes 2026-02-20T22:37:14.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6615917568 bytes 2026-02-20T22:37:14.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6650177536 bytes 2026-02-20T22:37:14.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6733765632 bytes 2026-02-20T22:37:14.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6782215168 bytes 2026-02-20T22:37:14.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6866501632 bytes 2026-02-20T22:37:14.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6897223680 bytes 2026-02-20T22:37:14.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6967495680 bytes 2026-02-20T22:37:14.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6995454976 bytes 2026-02-20T22:37:14.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7048969216 bytes 2026-02-20T22:37:14.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7088672768 bytes 2026-02-20T22:37:14.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7119388672 bytes 2026-02-20T22:37:14.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7166505984 bytes 2026-02-20T22:37:14.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7209456640 bytes 2026-02-20T22:37:14.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7246656512 bytes 2026-02-20T22:37:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7333765120 bytes 2026-02-20T22:37:14.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7370229760 bytes 2026-02-20T22:37:15.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7414709248 bytes 2026-02-20T22:37:15.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7426485248 bytes 2026-02-20T22:37:15.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7465927680 bytes 2026-02-20T22:37:15.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7470656512 bytes 2026-02-20T22:37:15.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7506851840 bytes 2026-02-20T22:37:15.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7511663616 bytes 2026-02-20T22:37:15.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7551099904 bytes 2026-02-20T22:37:15.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7560774656 bytes 2026-02-20T22:37:15.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7575444480 bytes 2026-02-20T22:37:15.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7606321152 bytes 2026-02-20T22:37:15.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7633886208 bytes 2026-02-20T22:37:15.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7642017792 bytes 2026-02-20T22:37:15.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7661257728 bytes 2026-02-20T22:37:15.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7712553984 bytes 2026-02-20T22:37:15.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7754045440 bytes 2026-02-20T22:37:15.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7829570560 bytes 2026-02-20T22:37:15.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7846300672 bytes 2026-02-20T22:37:15.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 7870121984 bytes 2026-02-20T22:37:15.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 7914673152 bytes 2026-02-20T22:37:15.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 7944368128 bytes 2026-02-20T22:37:16.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 7968026624 bytes 2026-02-20T22:37:16.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 7990117376 bytes 2026-02-20T22:37:16.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8031381504 bytes 2026-02-20T22:37:16.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8039707648 bytes 2026-02-20T22:37:16.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8102198272 bytes 2026-02-20T22:37:16.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8189350912 bytes 2026-02-20T22:37:16.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8233802752 bytes 2026-02-20T22:37:16.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8276722688 bytes 2026-02-20T22:37:16.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8336125952 bytes 2026-02-20T22:37:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8361621504 bytes 2026-02-20T22:37:16.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8430950400 bytes 2026-02-20T22:37:16.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8484564992 bytes 2026-02-20T22:37:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_9 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:16.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_9 128M 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:16.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2026-02-20T22:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:16.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2026-02-20T22:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:16.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2026-02-20T22:37:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:16.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_9': 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12b821699f82 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12b821699f82 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:16.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:16 2026 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:16 2026 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:16 2026 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: eae8172c-87a1-437f-8f5e-44b17669fdc4 2026-02-20T22:37:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_9 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:37:16.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:16.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=9 2026-02-20T22:37:16.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:16.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:17.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 96305152 bytes 2026-02-20T22:37:17.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 153318400 bytes 2026-02-20T22:37:17.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 207312896 bytes 2026-02-20T22:37:17.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 215845888 bytes 2026-02-20T22:37:17.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 332793856 bytes 2026-02-20T22:37:17.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 396180480 bytes 2026-02-20T22:37:17.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 460115968 bytes 2026-02-20T22:37:17.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 519692288 bytes 2026-02-20T22:37:17.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 571175936 bytes 2026-02-20T22:37:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 616957952 bytes 2026-02-20T22:37:17.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 680745984 bytes 2026-02-20T22:37:17.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 794219520 bytes 2026-02-20T22:37:17.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 884403200 bytes 2026-02-20T22:37:17.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 902209536 bytes 2026-02-20T22:37:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 939822080 bytes 2026-02-20T22:37:18.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 1021333504 bytes 2026-02-20T22:37:18.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 1078963200 bytes 2026-02-20T22:37:18.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 1124793344 bytes 2026-02-20T22:37:18.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 1170989056 bytes 2026-02-20T22:37:18.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1189955584 bytes 2026-02-20T22:37:18.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1229430784 bytes 2026-02-20T22:37:18.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1246054400 bytes 2026-02-20T22:37:18.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1284859904 bytes 2026-02-20T22:37:18.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1360968704 bytes 2026-02-20T22:37:18.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1430195200 bytes 2026-02-20T22:37:18.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1472985088 bytes 2026-02-20T22:37:18.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1493482496 bytes 2026-02-20T22:37:18.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1516837888 bytes 2026-02-20T22:37:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1547426816 bytes 2026-02-20T22:37:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1592511488 bytes 2026-02-20T22:37:18.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1621017600 bytes 2026-02-20T22:37:19.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1622908928 bytes 2026-02-20T22:37:19.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1669083136 bytes 2026-02-20T22:37:19.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1732025344 bytes 2026-02-20T22:37:19.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1771260928 bytes 2026-02-20T22:37:19.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1818408960 bytes 2026-02-20T22:37:19.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1849989120 bytes 2026-02-20T22:37:19.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1868431360 bytes 2026-02-20T22:37:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1899300864 bytes 2026-02-20T22:37:19.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1950019584 bytes 2026-02-20T22:37:19.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1976849408 bytes 2026-02-20T22:37:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2021642240 bytes 2026-02-20T22:37:19.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2078244864 bytes 2026-02-20T22:37:19.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2165411840 bytes 2026-02-20T22:37:19.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2225688576 bytes 2026-02-20T22:37:19.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2310609920 bytes 2026-02-20T22:37:19.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2365073408 bytes 2026-02-20T22:37:20.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2426628096 bytes 2026-02-20T22:37:20.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2460223488 bytes 2026-02-20T22:37:20.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2501812224 bytes 2026-02-20T22:37:20.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2512813056 bytes 2026-02-20T22:37:20.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2543435776 bytes 2026-02-20T22:37:20.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2576353280 bytes 2026-02-20T22:37:20.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2667226112 bytes 2026-02-20T22:37:20.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2722942976 bytes 2026-02-20T22:37:20.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2841884672 bytes 2026-02-20T22:37:20.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2883520512 bytes 2026-02-20T22:37:20.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2923827200 bytes 2026-02-20T22:37:20.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2963865600 bytes 2026-02-20T22:37:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 3008893952 bytes 2026-02-20T22:37:21.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 3080281088 bytes 2026-02-20T22:37:21.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 3130580992 bytes 2026-02-20T22:37:21.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 3155767296 bytes 2026-02-20T22:37:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3187275776 bytes 2026-02-20T22:37:21.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3307718656 bytes 2026-02-20T22:37:21.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3419561984 bytes 2026-02-20T22:37:21.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3458455552 bytes 2026-02-20T22:37:21.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3512583168 bytes 2026-02-20T22:37:21.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3560531968 bytes 2026-02-20T22:37:21.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3607079936 bytes 2026-02-20T22:37:21.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3631828992 bytes 2026-02-20T22:37:21.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3701066752 bytes 2026-02-20T22:37:21.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3715874816 bytes 2026-02-20T22:37:21.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3739820032 bytes 2026-02-20T22:37:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3741931520 bytes 2026-02-20T22:37:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3778820096 bytes 2026-02-20T22:37:22.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3847695360 bytes 2026-02-20T22:37:22.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3927049216 bytes 2026-02-20T22:37:22.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 4005193728 bytes 2026-02-20T22:37:22.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 4039907328 bytes 2026-02-20T22:37:22.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 4090833920 bytes 2026-02-20T22:37:22.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 4153582592 bytes 2026-02-20T22:37:22.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 4220312576 bytes 2026-02-20T22:37:22.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 4250321920 bytes 2026-02-20T22:37:22.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4322962432 bytes 2026-02-20T22:37:22.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4375583744 bytes 2026-02-20T22:37:22.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4393053184 bytes 2026-02-20T22:37:22.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4432831488 bytes 2026-02-20T22:37:22.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4485312512 bytes 2026-02-20T22:37:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4542154752 bytes 2026-02-20T22:37:23.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4592972800 bytes 2026-02-20T22:37:23.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4661954560 bytes 2026-02-20T22:37:23.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4697946112 bytes 2026-02-20T22:37:23.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4764206080 bytes 2026-02-20T22:37:23.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4797373440 bytes 2026-02-20T22:37:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4819594240 bytes 2026-02-20T22:37:23.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4856629248 bytes 2026-02-20T22:37:23.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4918888448 bytes 2026-02-20T22:37:23.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4961219584 bytes 2026-02-20T22:37:23.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 5022985216 bytes 2026-02-20T22:37:23.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 5056369664 bytes 2026-02-20T22:37:23.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 5094460416 bytes 2026-02-20T22:37:23.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 5146772480 bytes 2026-02-20T22:37:23.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 5169895424 bytes 2026-02-20T22:37:23.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 5220703232 bytes 2026-02-20T22:37:23.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 5307697152 bytes 2026-02-20T22:37:24.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 5355044864 bytes 2026-02-20T22:37:24.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 5369494528 bytes 2026-02-20T22:37:24.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 5407944704 bytes 2026-02-20T22:37:24.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5448464384 bytes 2026-02-20T22:37:24.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5529850880 bytes 2026-02-20T22:37:24.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5571013632 bytes 2026-02-20T22:37:24.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5620661248 bytes 2026-02-20T22:37:24.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5675151360 bytes 2026-02-20T22:37:24.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5686576128 bytes 2026-02-20T22:37:24.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5728317440 bytes 2026-02-20T22:37:24.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5801503744 bytes 2026-02-20T22:37:24.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5892452352 bytes 2026-02-20T22:37:24.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5977135104 bytes 2026-02-20T22:37:24.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 6004322304 bytes 2026-02-20T22:37:24.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 6012172288 bytes 2026-02-20T22:37:24.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 6104469504 bytes 2026-02-20T22:37:24.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 6139810816 bytes 2026-02-20T22:37:25.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 6223824896 bytes 2026-02-20T22:37:25.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 6315732992 bytes 2026-02-20T22:37:25.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 6375395328 bytes 2026-02-20T22:37:25.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 6393768960 bytes 2026-02-20T22:37:25.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 6442561536 bytes 2026-02-20T22:37:25.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 6450330624 bytes 2026-02-20T22:37:25.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6487866368 bytes 2026-02-20T22:37:25.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6525669376 bytes 2026-02-20T22:37:25.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6539728896 bytes 2026-02-20T22:37:25.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6599434240 bytes 2026-02-20T22:37:25.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6620660736 bytes 2026-02-20T22:37:25.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6624675840 bytes 2026-02-20T22:37:25.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6642222080 bytes 2026-02-20T22:37:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6669234176 bytes 2026-02-20T22:37:25.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6693897216 bytes 2026-02-20T22:37:25.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6728161280 bytes 2026-02-20T22:37:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:25.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_10 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:25.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_10 128M 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:25.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:25.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2026-02-20T22:37:25.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:25.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:25.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2026-02-20T22:37:26.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:26.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2026-02-20T22:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_10': 2026-02-20T22:37:26.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12c068298ce3 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12c068298ce3 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:25 2026 2026-02-20T22:37:26.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:25 2026 2026-02-20T22:37:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:25 2026 2026-02-20T22:37:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 21a4a0c9-3c5e-4e1d-a979-7e282caacbc3 2026-02-20T22:37:26.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:26.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_10 2026-02-20T22:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:37:26.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:26.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:37:26.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:26.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 46621696 bytes 2026-02-20T22:37:27.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 71408640 bytes 2026-02-20T22:37:27.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 126123008 bytes 2026-02-20T22:37:27.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 136327168 bytes 2026-02-20T22:37:27.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 174002176 bytes 2026-02-20T22:37:27.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 223679488 bytes 2026-02-20T22:37:27.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 364184576 bytes 2026-02-20T22:37:27.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 382403584 bytes 2026-02-20T22:37:27.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 394953728 bytes 2026-02-20T22:37:27.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 418593792 bytes 2026-02-20T22:37:27.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 475805696 bytes 2026-02-20T22:37:27.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 483688448 bytes 2026-02-20T22:37:27.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 492406784 bytes 2026-02-20T22:37:27.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 570104832 bytes 2026-02-20T22:37:27.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 578162688 bytes 2026-02-20T22:37:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 648773632 bytes 2026-02-20T22:37:27.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 719113216 bytes 2026-02-20T22:37:27.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 726348800 bytes 2026-02-20T22:37:28.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 759705600 bytes 2026-02-20T22:37:28.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 795048960 bytes 2026-02-20T22:37:28.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 838082560 bytes 2026-02-20T22:37:28.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 916252672 bytes 2026-02-20T22:37:28.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 983894016 bytes 2026-02-20T22:37:28.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1050307584 bytes 2026-02-20T22:37:28.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1050916864 bytes 2026-02-20T22:37:28.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1067721728 bytes 2026-02-20T22:37:28.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1098552320 bytes 2026-02-20T22:37:28.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1107366912 bytes 2026-02-20T22:37:28.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1157796864 bytes 2026-02-20T22:37:28.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1172776960 bytes 2026-02-20T22:37:28.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1201506304 bytes 2026-02-20T22:37:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1242739712 bytes 2026-02-20T22:37:29.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1282644992 bytes 2026-02-20T22:37:29.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1306251264 bytes 2026-02-20T22:37:29.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1334793216 bytes 2026-02-20T22:37:29.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1374183424 bytes 2026-02-20T22:37:29.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1401521152 bytes 2026-02-20T22:37:29.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1419575296 bytes 2026-02-20T22:37:29.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1424588800 bytes 2026-02-20T22:37:29.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1488715776 bytes 2026-02-20T22:37:29.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1539542016 bytes 2026-02-20T22:37:29.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1596838912 bytes 2026-02-20T22:37:29.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1637135360 bytes 2026-02-20T22:37:29.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1691108352 bytes 2026-02-20T22:37:29.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1741654016 bytes 2026-02-20T22:37:29.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1848901632 bytes 2026-02-20T22:37:29.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1878895616 bytes 2026-02-20T22:37:29.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1913830400 bytes 2026-02-20T22:37:29.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1967863808 bytes 2026-02-20T22:37:29.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1979703296 bytes 2026-02-20T22:37:30.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2004288512 bytes 2026-02-20T22:37:30.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2037724160 bytes 2026-02-20T22:37:30.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2074771456 bytes 2026-02-20T22:37:30.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2111764480 bytes 2026-02-20T22:37:30.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2120850432 bytes 2026-02-20T22:37:30.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2162439168 bytes 2026-02-20T22:37:30.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2226860032 bytes 2026-02-20T22:37:30.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2259203072 bytes 2026-02-20T22:37:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2298404864 bytes 2026-02-20T22:37:30.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2339924992 bytes 2026-02-20T22:37:30.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2373439488 bytes 2026-02-20T22:37:30.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2398011392 bytes 2026-02-20T22:37:30.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2434864128 bytes 2026-02-20T22:37:30.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2449618944 bytes 2026-02-20T22:37:30.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2484560896 bytes 2026-02-20T22:37:30.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2534733824 bytes 2026-02-20T22:37:30.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2624541696 bytes 2026-02-20T22:37:30.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2652827648 bytes 2026-02-20T22:37:31.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2725023744 bytes 2026-02-20T22:37:31.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2786319360 bytes 2026-02-20T22:37:31.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2858832896 bytes 2026-02-20T22:37:31.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2904356864 bytes 2026-02-20T22:37:31.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2929418240 bytes 2026-02-20T22:37:31.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2970859520 bytes 2026-02-20T22:37:31.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3009746944 bytes 2026-02-20T22:37:31.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3010374656 bytes 2026-02-20T22:37:31.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3040281600 bytes 2026-02-20T22:37:31.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3132625920 bytes 2026-02-20T22:37:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3177541632 bytes 2026-02-20T22:37:31.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3213670400 bytes 2026-02-20T22:37:31.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3269452800 bytes 2026-02-20T22:37:31.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3294817280 bytes 2026-02-20T22:37:31.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3343566848 bytes 2026-02-20T22:37:31.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3344912384 bytes 2026-02-20T22:37:32.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3369556992 bytes 2026-02-20T22:37:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3425595392 bytes 2026-02-20T22:37:32.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3496881152 bytes 2026-02-20T22:37:32.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3514080256 bytes 2026-02-20T22:37:32.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3550803968 bytes 2026-02-20T22:37:32.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3592582144 bytes 2026-02-20T22:37:32.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3655152640 bytes 2026-02-20T22:37:32.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3655772160 bytes 2026-02-20T22:37:32.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3789524992 bytes 2026-02-20T22:37:32.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3797431296 bytes 2026-02-20T22:37:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3831045120 bytes 2026-02-20T22:37:32.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3859722240 bytes 2026-02-20T22:37:32.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3932300288 bytes 2026-02-20T22:37:32.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3955792896 bytes 2026-02-20T22:37:32.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3986640896 bytes 2026-02-20T22:37:32.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3990981632 bytes 2026-02-20T22:37:32.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4015881216 bytes 2026-02-20T22:37:33.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4103127040 bytes 2026-02-20T22:37:33.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4141711360 bytes 2026-02-20T22:37:33.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4156539904 bytes 2026-02-20T22:37:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4200040448 bytes 2026-02-20T22:37:33.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4248012800 bytes 2026-02-20T22:37:33.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4313802752 bytes 2026-02-20T22:37:33.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4327713792 bytes 2026-02-20T22:37:33.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4389450752 bytes 2026-02-20T22:37:33.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4455296000 bytes 2026-02-20T22:37:33.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4480747520 bytes 2026-02-20T22:37:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4519869440 bytes 2026-02-20T22:37:33.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4548519936 bytes 2026-02-20T22:37:33.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4556702720 bytes 2026-02-20T22:37:33.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4596052992 bytes 2026-02-20T22:37:33.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4675464192 bytes 2026-02-20T22:37:33.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4723751936 bytes 2026-02-20T22:37:33.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4779243520 bytes 2026-02-20T22:37:33.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4798312448 bytes 2026-02-20T22:37:34.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4862279680 bytes 2026-02-20T22:37:34.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4911747072 bytes 2026-02-20T22:37:34.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4934438912 bytes 2026-02-20T22:37:34.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4990642176 bytes 2026-02-20T22:37:34.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5006857216 bytes 2026-02-20T22:37:34.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5056759808 bytes 2026-02-20T22:37:34.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5065800704 bytes 2026-02-20T22:37:34.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5103519744 bytes 2026-02-20T22:37:34.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5141792768 bytes 2026-02-20T22:37:34.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5206418432 bytes 2026-02-20T22:37:34.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5237931008 bytes 2026-02-20T22:37:34.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5251735552 bytes 2026-02-20T22:37:34.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5267387392 bytes 2026-02-20T22:37:34.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5349803008 bytes 2026-02-20T22:37:34.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5362281472 bytes 2026-02-20T22:37:34.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5384055808 bytes 2026-02-20T22:37:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5481257984 bytes 2026-02-20T22:37:34.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5531116544 bytes 2026-02-20T22:37:35.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5555579904 bytes 2026-02-20T22:37:35.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5567301632 bytes 2026-02-20T22:37:35.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5621608448 bytes 2026-02-20T22:37:35.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5663761408 bytes 2026-02-20T22:37:35.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5706431488 bytes 2026-02-20T22:37:35.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5781828608 bytes 2026-02-20T22:37:35.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5817239552 bytes 2026-02-20T22:37:35.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5846515712 bytes 2026-02-20T22:37:35.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5886524416 bytes 2026-02-20T22:37:35.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5910667264 bytes 2026-02-20T22:37:35.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5988324352 bytes 2026-02-20T22:37:35.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6014738432 bytes 2026-02-20T22:37:35.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6076496896 bytes 2026-02-20T22:37:35.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6112361472 bytes 2026-02-20T22:37:35.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6149856256 bytes 2026-02-20T22:37:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6179844096 bytes 2026-02-20T22:37:35.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6205557760 bytes 2026-02-20T22:37:36.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6234737664 bytes 2026-02-20T22:37:36.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6292458496 bytes 2026-02-20T22:37:36.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6344111104 bytes 2026-02-20T22:37:36.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6375948288 bytes 2026-02-20T22:37:36.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6430236672 bytes 2026-02-20T22:37:36.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6464199680 bytes 2026-02-20T22:37:36.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6477627392 bytes 2026-02-20T22:37:36.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6522025984 bytes 2026-02-20T22:37:36.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6533005312 bytes 2026-02-20T22:37:36.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6610481152 bytes 2026-02-20T22:37:36.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6639832064 bytes 2026-02-20T22:37:36.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6674242560 bytes 2026-02-20T22:37:36.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6719073280 bytes 2026-02-20T22:37:36.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6744056832 bytes 2026-02-20T22:37:37.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6808232960 bytes 2026-02-20T22:37:37.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 6852806656 bytes 2026-02-20T22:37:37.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 6901308416 bytes 2026-02-20T22:37:37.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 6945569792 bytes 2026-02-20T22:37:37.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 6992967680 bytes 2026-02-20T22:37:37.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7048889344 bytes 2026-02-20T22:37:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7061920768 bytes 2026-02-20T22:37:37.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7135029248 bytes 2026-02-20T22:37:37.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7135625216 bytes 2026-02-20T22:37:37.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7184464896 bytes 2026-02-20T22:37:37.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7210897408 bytes 2026-02-20T22:37:37.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7254519808 bytes 2026-02-20T22:37:37.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7329168384 bytes 2026-02-20T22:37:37.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7366992896 bytes 2026-02-20T22:37:37.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7386066944 bytes 2026-02-20T22:37:37.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7430100992 bytes 2026-02-20T22:37:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7465048064 bytes 2026-02-20T22:37:38.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7525909504 bytes 2026-02-20T22:37:38.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7615084544 bytes 2026-02-20T22:37:38.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7676224512 bytes 2026-02-20T22:37:38.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7778664448 bytes 2026-02-20T22:37:38.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7867019264 bytes 2026-02-20T22:37:38.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7894021120 bytes 2026-02-20T22:37:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7933018112 bytes 2026-02-20T22:37:38.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7938716672 bytes 2026-02-20T22:37:38.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7993589760 bytes 2026-02-20T22:37:38.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8042356736 bytes 2026-02-20T22:37:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8074296320 bytes 2026-02-20T22:37:38.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8168875008 bytes 2026-02-20T22:37:38.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8193514496 bytes 2026-02-20T22:37:38.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8236162048 bytes 2026-02-20T22:37:38.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8262550528 bytes 2026-02-20T22:37:39.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8323109888 bytes 2026-02-20T22:37:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8342953984 bytes 2026-02-20T22:37:39.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8369740800 bytes 2026-02-20T22:37:39.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8382861312 bytes 2026-02-20T22:37:39.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8422700032 bytes 2026-02-20T22:37:39.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8484666368 bytes 2026-02-20T22:37:39.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8500595712 bytes 2026-02-20T22:37:39.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8524903424 bytes 2026-02-20T22:37:39.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8556355584 bytes 2026-02-20T22:37:39.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8587769856 bytes 2026-02-20T22:37:39.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8611446784 bytes 2026-02-20T22:37:39.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 8642189312 bytes 2026-02-20T22:37:39.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 8653502464 bytes 2026-02-20T22:37:39.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 8713476096 bytes 2026-02-20T22:37:39.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 8729587712 bytes 2026-02-20T22:37:39.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 8761839616 bytes 2026-02-20T22:37:39.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 8806537216 bytes 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_11 2026-02-20T22:37:39.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_11 128M 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:39.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2026-02-20T22:37:39.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:39.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:37:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:39.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2026-02-20T22:37:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:40.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_11': 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12cafb848533 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12cafb848533 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:39 2026 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:39 2026 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:39 2026 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: c1afeba9-ed53-4844-9dd9-34f5c5ee4c22 2026-02-20T22:37:40.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_11 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:37:40.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:40.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:37:40.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:40.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:41.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 19012608 bytes 2026-02-20T22:37:41.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 97188864 bytes 2026-02-20T22:37:41.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 121465856 bytes 2026-02-20T22:37:41.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 155415552 bytes 2026-02-20T22:37:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 199632896 bytes 2026-02-20T22:37:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 212063232 bytes 2026-02-20T22:37:41.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 252382208 bytes 2026-02-20T22:37:41.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 279695360 bytes 2026-02-20T22:37:41.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 355081216 bytes 2026-02-20T22:37:41.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 400420864 bytes 2026-02-20T22:37:41.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 427611136 bytes 2026-02-20T22:37:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 504710144 bytes 2026-02-20T22:37:41.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 566374400 bytes 2026-02-20T22:37:41.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 658341888 bytes 2026-02-20T22:37:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 685163520 bytes 2026-02-20T22:37:41.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 715978752 bytes 2026-02-20T22:37:41.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 789225472 bytes 2026-02-20T22:37:42.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 823388160 bytes 2026-02-20T22:37:42.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 890018816 bytes 2026-02-20T22:37:42.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 890620928 bytes 2026-02-20T22:37:42.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 946755584 bytes 2026-02-20T22:37:42.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 968151040 bytes 2026-02-20T22:37:42.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1023624192 bytes 2026-02-20T22:37:42.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1055147008 bytes 2026-02-20T22:37:42.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1105425408 bytes 2026-02-20T22:37:42.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1139852288 bytes 2026-02-20T22:37:42.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1190379520 bytes 2026-02-20T22:37:42.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1267783680 bytes 2026-02-20T22:37:42.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1303388160 bytes 2026-02-20T22:37:42.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1352615936 bytes 2026-02-20T22:37:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1353228288 bytes 2026-02-20T22:37:42.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1446681600 bytes 2026-02-20T22:37:42.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1497477120 bytes 2026-02-20T22:37:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1531956224 bytes 2026-02-20T22:37:43.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1567373312 bytes 2026-02-20T22:37:43.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1580068864 bytes 2026-02-20T22:37:43.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1707329536 bytes 2026-02-20T22:37:43.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1707940864 bytes 2026-02-20T22:37:43.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1732910080 bytes 2026-02-20T22:37:43.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1840142336 bytes 2026-02-20T22:37:43.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1857511424 bytes 2026-02-20T22:37:43.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1898632192 bytes 2026-02-20T22:37:43.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1921257472 bytes 2026-02-20T22:37:43.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1965845504 bytes 2026-02-20T22:37:43.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1975840768 bytes 2026-02-20T22:37:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1989285888 bytes 2026-02-20T22:37:43.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2070749184 bytes 2026-02-20T22:37:43.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2093075456 bytes 2026-02-20T22:37:43.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2093693952 bytes 2026-02-20T22:37:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2104987648 bytes 2026-02-20T22:37:43.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2210909184 bytes 2026-02-20T22:37:44.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2269763584 bytes 2026-02-20T22:37:44.024 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.45068366766449264) > weight (0.25) 2026-02-20T22:37:44.024 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7034380405151358) > weight (0.25) 2026-02-20T22:37:44.024 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2026-02-20T22:37:44.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2026-02-20T22:37:44.025 INFO:tasks.rbd_mirror_thrash:waiting for 6.972429631591216 secs before reviving daemons 2026-02-20T22:37:44.026 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:37:44.018+0000 7fdd45e4b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 19538) UID: 1000 2026-02-20T22:37:44.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2350979072 bytes 2026-02-20T22:37:44.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2447280128 bytes 2026-02-20T22:37:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2506344448 bytes 2026-02-20T22:37:44.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2558153728 bytes 2026-02-20T22:37:44.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2625014784 bytes 2026-02-20T22:37:44.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2736783360 bytes 2026-02-20T22:37:44.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2791487488 bytes 2026-02-20T22:37:44.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2844505088 bytes 2026-02-20T22:37:44.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2895127552 bytes 2026-02-20T22:37:44.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2939082752 bytes 2026-02-20T22:37:44.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2981826560 bytes 2026-02-20T22:37:44.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3067492352 bytes 2026-02-20T22:37:44.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3116040192 bytes 2026-02-20T22:37:44.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3126038528 bytes 2026-02-20T22:37:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3168105472 bytes 2026-02-20T22:37:45.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3282318336 bytes 2026-02-20T22:37:45.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3282929664 bytes 2026-02-20T22:37:45.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3374196736 bytes 2026-02-20T22:37:45.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3424414720 bytes 2026-02-20T22:37:45.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3445544960 bytes 2026-02-20T22:37:45.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3494894592 bytes 2026-02-20T22:37:45.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3495512064 bytes 2026-02-20T22:37:45.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3515319296 bytes 2026-02-20T22:37:45.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3552382976 bytes 2026-02-20T22:37:45.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3626568704 bytes 2026-02-20T22:37:45.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3675032576 bytes 2026-02-20T22:37:45.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3694757888 bytes 2026-02-20T22:37:45.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3717524480 bytes 2026-02-20T22:37:45.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3743878144 bytes 2026-02-20T22:37:45.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3795094528 bytes 2026-02-20T22:37:45.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3819272192 bytes 2026-02-20T22:37:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:45.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_12 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:45.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_12 128M 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:45.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2026-02-20T22:37:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:46.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2026-02-20T22:37:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_12': 2026-02-20T22:37:46.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12d4465823f6 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12d4465823f6 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:45 2026 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:45 2026 2026-02-20T22:37:46.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:45 2026 2026-02-20T22:37:46.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:46.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:46.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 97e62ce9-2919-43d0-b90f-ef29433498b6 2026-02-20T22:37:46.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:46.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:46.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_12 2026-02-20T22:37:46.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:46.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:37:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:46.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=8 2026-02-20T22:37:46.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:46.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:47.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 60116992 bytes 2026-02-20T22:37:47.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 141452288 bytes 2026-02-20T22:37:47.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 162896896 bytes 2026-02-20T22:37:47.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 189998080 bytes 2026-02-20T22:37:47.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 271135744 bytes 2026-02-20T22:37:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 330139648 bytes 2026-02-20T22:37:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 336491520 bytes 2026-02-20T22:37:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 372060160 bytes 2026-02-20T22:37:47.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 379800576 bytes 2026-02-20T22:37:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 406068224 bytes 2026-02-20T22:37:47.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 426420224 bytes 2026-02-20T22:37:47.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 472111104 bytes 2026-02-20T22:37:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 522737664 bytes 2026-02-20T22:37:47.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 571421696 bytes 2026-02-20T22:37:47.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 674659328 bytes 2026-02-20T22:37:47.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 721039360 bytes 2026-02-20T22:37:47.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 768346112 bytes 2026-02-20T22:37:48.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 838679552 bytes 2026-02-20T22:37:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 865096704 bytes 2026-02-20T22:37:48.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 987033600 bytes 2026-02-20T22:37:48.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1007870976 bytes 2026-02-20T22:37:48.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1069282304 bytes 2026-02-20T22:37:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1122909184 bytes 2026-02-20T22:37:48.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1134402560 bytes 2026-02-20T22:37:48.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1227330560 bytes 2026-02-20T22:37:48.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1264574464 bytes 2026-02-20T22:37:48.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1309784064 bytes 2026-02-20T22:37:48.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1351209984 bytes 2026-02-20T22:37:48.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1383314432 bytes 2026-02-20T22:37:48.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1400461312 bytes 2026-02-20T22:37:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1466639360 bytes 2026-02-20T22:37:48.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1501275136 bytes 2026-02-20T22:37:48.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1556862976 bytes 2026-02-20T22:37:49.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1646553088 bytes 2026-02-20T22:37:49.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1673229312 bytes 2026-02-20T22:37:49.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1730245632 bytes 2026-02-20T22:37:49.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1800414208 bytes 2026-02-20T22:37:49.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1871308800 bytes 2026-02-20T22:37:49.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1876865024 bytes 2026-02-20T22:37:49.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1877480448 bytes 2026-02-20T22:37:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1885774848 bytes 2026-02-20T22:37:49.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1976518656 bytes 2026-02-20T22:37:49.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2006565888 bytes 2026-02-20T22:37:49.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2054560768 bytes 2026-02-20T22:37:49.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2078649344 bytes 2026-02-20T22:37:49.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2182980608 bytes 2026-02-20T22:37:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2202096640 bytes 2026-02-20T22:37:49.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2202713088 bytes 2026-02-20T22:37:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2240734208 bytes 2026-02-20T22:37:49.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2266967040 bytes 2026-02-20T22:37:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2338551808 bytes 2026-02-20T22:37:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2393919488 bytes 2026-02-20T22:37:50.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2432123904 bytes 2026-02-20T22:37:50.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2481060864 bytes 2026-02-20T22:37:50.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2487682048 bytes 2026-02-20T22:37:50.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2520087552 bytes 2026-02-20T22:37:50.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2520686592 bytes 2026-02-20T22:37:50.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2560578560 bytes 2026-02-20T22:37:50.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2565917696 bytes 2026-02-20T22:37:50.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2630347776 bytes 2026-02-20T22:37:50.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2694538240 bytes 2026-02-20T22:37:50.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2712858624 bytes 2026-02-20T22:37:50.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2777622528 bytes 2026-02-20T22:37:50.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2825697280 bytes 2026-02-20T22:37:50.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2880115712 bytes 2026-02-20T22:37:50.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2900832256 bytes 2026-02-20T22:37:50.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2929546240 bytes 2026-02-20T22:37:50.998 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2026-02-20T22:37:50.998 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:37:50.999 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2026-02-20T22:37:50.999 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-20T22:37:50.999 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-20T22:37:51.002 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-20T22:37:51.002 INFO:tasks.rbd_mirror_thrash:waiting for 118.86707127203402 secs before thrashing 2026-02-20T22:37:51.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2979899392 bytes 2026-02-20T22:37:51.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2996662272 bytes 2026-02-20T22:37:51.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3041761280 bytes 2026-02-20T22:37:51.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3109293056 bytes 2026-02-20T22:37:51.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3159002112 bytes 2026-02-20T22:37:51.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3203249152 bytes 2026-02-20T22:37:51.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3251985408 bytes 2026-02-20T22:37:51.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3269957632 bytes 2026-02-20T22:37:51.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3290909696 bytes 2026-02-20T22:37:51.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3366135808 bytes 2026-02-20T22:37:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3410982912 bytes 2026-02-20T22:37:51.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3419222016 bytes 2026-02-20T22:37:51.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3490955264 bytes 2026-02-20T22:37:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3518180352 bytes 2026-02-20T22:37:51.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3545353216 bytes 2026-02-20T22:37:51.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3560404992 bytes 2026-02-20T22:37:51.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3591570432 bytes 2026-02-20T22:37:51.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3611853824 bytes 2026-02-20T22:37:52.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3653206016 bytes 2026-02-20T22:37:52.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3698788352 bytes 2026-02-20T22:37:52.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3750904832 bytes 2026-02-20T22:37:52.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3812073472 bytes 2026-02-20T22:37:52.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3833436160 bytes 2026-02-20T22:37:52.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3885950976 bytes 2026-02-20T22:37:52.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3886573568 bytes 2026-02-20T22:37:52.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3937830912 bytes 2026-02-20T22:37:52.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3966500864 bytes 2026-02-20T22:37:52.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4001390592 bytes 2026-02-20T22:37:52.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4054576128 bytes 2026-02-20T22:37:52.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4084845568 bytes 2026-02-20T22:37:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4137255936 bytes 2026-02-20T22:37:52.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4196692992 bytes 2026-02-20T22:37:52.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4239620096 bytes 2026-02-20T22:37:52.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4306292736 bytes 2026-02-20T22:37:52.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4365883392 bytes 2026-02-20T22:37:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4412849152 bytes 2026-02-20T22:37:53.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4423572480 bytes 2026-02-20T22:37:53.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4461993984 bytes 2026-02-20T22:37:53.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4497321984 bytes 2026-02-20T22:37:53.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4571804672 bytes 2026-02-20T22:37:53.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4605367296 bytes 2026-02-20T22:37:53.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4673255424 bytes 2026-02-20T22:37:53.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4728305664 bytes 2026-02-20T22:37:53.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4804101120 bytes 2026-02-20T22:37:53.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4881799168 bytes 2026-02-20T22:37:53.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4924837888 bytes 2026-02-20T22:37:53.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4940203008 bytes 2026-02-20T22:37:53.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4990479360 bytes 2026-02-20T22:37:53.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5056927744 bytes 2026-02-20T22:37:53.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5140124672 bytes 2026-02-20T22:37:54.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5192674304 bytes 2026-02-20T22:37:54.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5269521408 bytes 2026-02-20T22:37:54.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5273582592 bytes 2026-02-20T22:37:54.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5353164800 bytes 2026-02-20T22:37:54.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5408797696 bytes 2026-02-20T22:37:54.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5436448768 bytes 2026-02-20T22:37:54.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5472210944 bytes 2026-02-20T22:37:54.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5500152832 bytes 2026-02-20T22:37:54.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5565437952 bytes 2026-02-20T22:37:54.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5587298304 bytes 2026-02-20T22:37:54.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5650300928 bytes 2026-02-20T22:37:54.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5696726016 bytes 2026-02-20T22:37:54.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5760842752 bytes 2026-02-20T22:37:54.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5879790592 bytes 2026-02-20T22:37:54.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5919587328 bytes 2026-02-20T22:37:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:37:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:37:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:37:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:54.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_13 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_13 128M 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:37:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:37:54.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2026-02-20T22:37:55.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:37:55.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2026-02-20T22:37:55.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_13': 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12e6ac6df3a0 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12e6ac6df3a0 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:37:55.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:54 2026 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:54 2026 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:54 2026 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 63774277-8720-40a9-a26f-36b0c3afe1c0 2026-02-20T22:37:55.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:37:55.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:37:55.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_13 2026-02-20T22:37:55.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:37:55.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:37:55.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:37:55.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:37:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:37:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:37:55.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:37:55.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 51915776 bytes 2026-02-20T22:37:55.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 76922880 bytes 2026-02-20T22:37:55.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 107578368 bytes 2026-02-20T22:37:55.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 143759360 bytes 2026-02-20T22:37:55.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 192395264 bytes 2026-02-20T22:37:56.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 293396480 bytes 2026-02-20T22:37:56.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 310458368 bytes 2026-02-20T22:37:56.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 347406336 bytes 2026-02-20T22:37:56.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 374247424 bytes 2026-02-20T22:37:56.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 410625024 bytes 2026-02-20T22:37:56.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 500630528 bytes 2026-02-20T22:37:56.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 552511488 bytes 2026-02-20T22:37:56.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 561871872 bytes 2026-02-20T22:37:56.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 612794368 bytes 2026-02-20T22:37:56.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 629111808 bytes 2026-02-20T22:37:56.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 650430464 bytes 2026-02-20T22:37:56.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 767239168 bytes 2026-02-20T22:37:56.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 816099328 bytes 2026-02-20T22:37:56.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 816726016 bytes 2026-02-20T22:37:56.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 864988160 bytes 2026-02-20T22:37:56.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 891176960 bytes 2026-02-20T22:37:57.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 976278528 bytes 2026-02-20T22:37:57.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 999932928 bytes 2026-02-20T22:37:57.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1038522368 bytes 2026-02-20T22:37:57.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1085347840 bytes 2026-02-20T22:37:57.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1133242368 bytes 2026-02-20T22:37:57.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1162364928 bytes 2026-02-20T22:37:57.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1185208320 bytes 2026-02-20T22:37:57.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1265359872 bytes 2026-02-20T22:37:57.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1271912448 bytes 2026-02-20T22:37:57.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1300119552 bytes 2026-02-20T22:37:57.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1350632448 bytes 2026-02-20T22:37:57.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1456249856 bytes 2026-02-20T22:37:57.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1513126912 bytes 2026-02-20T22:37:57.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1597749248 bytes 2026-02-20T22:37:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1624353792 bytes 2026-02-20T22:37:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1649825792 bytes 2026-02-20T22:37:58.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1729998848 bytes 2026-02-20T22:37:58.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1770152960 bytes 2026-02-20T22:37:58.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1827824640 bytes 2026-02-20T22:37:58.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1863388160 bytes 2026-02-20T22:37:58.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1915419648 bytes 2026-02-20T22:37:58.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1931893760 bytes 2026-02-20T22:37:58.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1999874048 bytes 2026-02-20T22:37:58.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2037562368 bytes 2026-02-20T22:37:58.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2054576128 bytes 2026-02-20T22:37:58.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2113610752 bytes 2026-02-20T22:37:58.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2185413632 bytes 2026-02-20T22:37:58.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2201253888 bytes 2026-02-20T22:37:58.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2216731648 bytes 2026-02-20T22:37:58.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2289058816 bytes 2026-02-20T22:37:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2395689984 bytes 2026-02-20T22:37:58.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2447990784 bytes 2026-02-20T22:37:59.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2488832000 bytes 2026-02-20T22:37:59.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2503839744 bytes 2026-02-20T22:37:59.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2533212160 bytes 2026-02-20T22:37:59.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2605184000 bytes 2026-02-20T22:37:59.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2605803520 bytes 2026-02-20T22:37:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2689173504 bytes 2026-02-20T22:37:59.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2770406400 bytes 2026-02-20T22:37:59.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2785444864 bytes 2026-02-20T22:37:59.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2849944576 bytes 2026-02-20T22:37:59.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2872408064 bytes 2026-02-20T22:37:59.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2894388224 bytes 2026-02-20T22:37:59.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2904249344 bytes 2026-02-20T22:37:59.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2944025600 bytes 2026-02-20T22:37:59.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3048323072 bytes 2026-02-20T22:37:59.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3096881152 bytes 2026-02-20T22:37:59.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3097513984 bytes 2026-02-20T22:37:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3126288384 bytes 2026-02-20T22:38:00.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3184389120 bytes 2026-02-20T22:38:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3200734208 bytes 2026-02-20T22:38:00.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3209481216 bytes 2026-02-20T22:38:00.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3249689600 bytes 2026-02-20T22:38:00.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3288060928 bytes 2026-02-20T22:38:00.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3346053120 bytes 2026-02-20T22:38:00.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3387237376 bytes 2026-02-20T22:38:00.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3426910208 bytes 2026-02-20T22:38:00.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3465970688 bytes 2026-02-20T22:38:00.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3532888064 bytes 2026-02-20T22:38:00.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3569603584 bytes 2026-02-20T22:38:00.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3584835584 bytes 2026-02-20T22:38:00.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3621568512 bytes 2026-02-20T22:38:00.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3692186624 bytes 2026-02-20T22:38:00.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3773553664 bytes 2026-02-20T22:38:00.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3806984192 bytes 2026-02-20T22:38:00.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3824382976 bytes 2026-02-20T22:38:00.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3863886848 bytes 2026-02-20T22:38:01.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3964820480 bytes 2026-02-20T22:38:01.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4005405696 bytes 2026-02-20T22:38:01.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4065477632 bytes 2026-02-20T22:38:01.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4098527232 bytes 2026-02-20T22:38:01.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4107890688 bytes 2026-02-20T22:38:01.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4134423552 bytes 2026-02-20T22:38:01.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4154385408 bytes 2026-02-20T22:38:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4222051328 bytes 2026-02-20T22:38:01.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4269544448 bytes 2026-02-20T22:38:01.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4331321344 bytes 2026-02-20T22:38:01.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4406642688 bytes 2026-02-20T22:38:01.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4441513984 bytes 2026-02-20T22:38:01.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4471804928 bytes 2026-02-20T22:38:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4541962240 bytes 2026-02-20T22:38:02.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4644658176 bytes 2026-02-20T22:38:02.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4669704192 bytes 2026-02-20T22:38:02.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4714812416 bytes 2026-02-20T22:38:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4726194176 bytes 2026-02-20T22:38:02.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4788240384 bytes 2026-02-20T22:38:02.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4851815424 bytes 2026-02-20T22:38:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4882127872 bytes 2026-02-20T22:38:02.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4905458688 bytes 2026-02-20T22:38:02.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4934352896 bytes 2026-02-20T22:38:02.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4957365248 bytes 2026-02-20T22:38:02.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4962506752 bytes 2026-02-20T22:38:02.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4995739648 bytes 2026-02-20T22:38:02.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4996353024 bytes 2026-02-20T22:38:02.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5054770176 bytes 2026-02-20T22:38:02.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5063280640 bytes 2026-02-20T22:38:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5133201408 bytes 2026-02-20T22:38:02.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5200492544 bytes 2026-02-20T22:38:02.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5248592896 bytes 2026-02-20T22:38:03.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5307735040 bytes 2026-02-20T22:38:03.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5325058048 bytes 2026-02-20T22:38:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5363463168 bytes 2026-02-20T22:38:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5378196480 bytes 2026-02-20T22:38:03.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5425617920 bytes 2026-02-20T22:38:03.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5441244160 bytes 2026-02-20T22:38:03.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5477512192 bytes 2026-02-20T22:38:03.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5549661184 bytes 2026-02-20T22:38:03.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5559171072 bytes 2026-02-20T22:38:03.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5593096192 bytes 2026-02-20T22:38:03.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5633189888 bytes 2026-02-20T22:38:03.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5654367232 bytes 2026-02-20T22:38:03.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5708540928 bytes 2026-02-20T22:38:03.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5741075456 bytes 2026-02-20T22:38:03.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5766672384 bytes 2026-02-20T22:38:03.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5835509760 bytes 2026-02-20T22:38:03.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5877250048 bytes 2026-02-20T22:38:03.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5909819392 bytes 2026-02-20T22:38:03.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5925245952 bytes 2026-02-20T22:38:04.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5987016704 bytes 2026-02-20T22:38:04.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6046232576 bytes 2026-02-20T22:38:04.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6076556288 bytes 2026-02-20T22:38:04.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6077167616 bytes 2026-02-20T22:38:04.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6101799936 bytes 2026-02-20T22:38:04.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6104679424 bytes 2026-02-20T22:38:04.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6141597696 bytes 2026-02-20T22:38:04.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6176709632 bytes 2026-02-20T22:38:04.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6191345664 bytes 2026-02-20T22:38:04.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6241306624 bytes 2026-02-20T22:38:04.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6262508544 bytes 2026-02-20T22:38:04.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6381263872 bytes 2026-02-20T22:38:04.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6440407040 bytes 2026-02-20T22:38:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6441027584 bytes 2026-02-20T22:38:04.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6482478080 bytes 2026-02-20T22:38:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6513724416 bytes 2026-02-20T22:38:04.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6581005312 bytes 2026-02-20T22:38:04.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6581620736 bytes 2026-02-20T22:38:05.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6620189696 bytes 2026-02-20T22:38:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6626361344 bytes 2026-02-20T22:38:05.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6673494016 bytes 2026-02-20T22:38:05.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6710594560 bytes 2026-02-20T22:38:05.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6812780544 bytes 2026-02-20T22:38:05.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6885834752 bytes 2026-02-20T22:38:05.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6944948224 bytes 2026-02-20T22:38:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7034081280 bytes 2026-02-20T22:38:05.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7050451968 bytes 2026-02-20T22:38:05.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7056923648 bytes 2026-02-20T22:38:05.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7117480960 bytes 2026-02-20T22:38:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7183844352 bytes 2026-02-20T22:38:06.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7271388160 bytes 2026-02-20T22:38:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7294167040 bytes 2026-02-20T22:38:06.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7320594432 bytes 2026-02-20T22:38:06.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7350114304 bytes 2026-02-20T22:38:06.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7414213632 bytes 2026-02-20T22:38:06.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7481254912 bytes 2026-02-20T22:38:06.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7562238976 bytes 2026-02-20T22:38:06.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7562852352 bytes 2026-02-20T22:38:06.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7576042496 bytes 2026-02-20T22:38:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7642558464 bytes 2026-02-20T22:38:06.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7670775808 bytes 2026-02-20T22:38:06.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7721974784 bytes 2026-02-20T22:38:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7784701952 bytes 2026-02-20T22:38:06.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7833174016 bytes 2026-02-20T22:38:06.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7880547328 bytes 2026-02-20T22:38:06.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7906581504 bytes 2026-02-20T22:38:06.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7969717248 bytes 2026-02-20T22:38:07.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8010090496 bytes 2026-02-20T22:38:07.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8025326592 bytes 2026-02-20T22:38:07.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8092898304 bytes 2026-02-20T22:38:07.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8137101312 bytes 2026-02-20T22:38:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8149085184 bytes 2026-02-20T22:38:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8320587776 bytes 2026-02-20T22:38:07.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8361469952 bytes 2026-02-20T22:38:07.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8395427840 bytes 2026-02-20T22:38:07.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8434771968 bytes 2026-02-20T22:38:07.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8464599040 bytes 2026-02-20T22:38:07.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8523497472 bytes 2026-02-20T22:38:07.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8554573824 bytes 2026-02-20T22:38:07.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8576442368 bytes 2026-02-20T22:38:07.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8624882688 bytes 2026-02-20T22:38:07.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8649478144 bytes 2026-02-20T22:38:07.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8700359680 bytes 2026-02-20T22:38:08.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8768954368 bytes 2026-02-20T22:38:08.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8822575104 bytes 2026-02-20T22:38:08.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8891357184 bytes 2026-02-20T22:38:08.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8936208384 bytes 2026-02-20T22:38:08.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9034652672 bytes 2026-02-20T22:38:08.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9097373696 bytes 2026-02-20T22:38:08.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9134452736 bytes 2026-02-20T22:38:08.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9201045504 bytes 2026-02-20T22:38:08.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9204757504 bytes 2026-02-20T22:38:08.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9307512832 bytes 2026-02-20T22:38:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9351325696 bytes 2026-02-20T22:38:08.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9369927680 bytes 2026-02-20T22:38:08.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9388731392 bytes 2026-02-20T22:38:08.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9440109568 bytes 2026-02-20T22:38:08.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9473007616 bytes 2026-02-20T22:38:08.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9494246400 bytes 2026-02-20T22:38:09.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9509267456 bytes 2026-02-20T22:38:09.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9529482240 bytes 2026-02-20T22:38:09.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9547716608 bytes 2026-02-20T22:38:09.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9588028416 bytes 2026-02-20T22:38:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9628297216 bytes 2026-02-20T22:38:09.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9650390016 bytes 2026-02-20T22:38:09.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9702132736 bytes 2026-02-20T22:38:09.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9746825216 bytes 2026-02-20T22:38:09.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 9794026496 bytes 2026-02-20T22:38:09.488 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:38:09.488 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:38:09.488 INFO:tasks.rbd_mirror_thrash:waiting for 0.8494316172032135 secs before reviving daemons 2026-02-20T22:38:09.489 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:38:09.482+0000 7f64e9516640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 27418) UID: 1000 2026-02-20T22:38:09.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 9828701184 bytes 2026-02-20T22:38:09.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 9863074816 bytes 2026-02-20T22:38:09.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 9916239872 bytes 2026-02-20T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:09.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_14 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:09.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_14 128M 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:09.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:09.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:38:09.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:09.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2026-02-20T22:38:10.338 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:38:10.338 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:38:10.338 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:38:10.339 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:38:10.339 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:38:10.341 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:38:10.341 INFO:tasks.rbd_mirror_thrash:waiting for 112.52874136149453 secs before thrashing 2026-02-20T22:38:10.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:10.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2026-02-20T22:38:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_14': 2026-02-20T22:38:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:10.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12f29559b8fc 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12f29559b8fc 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:09 2026 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:09 2026 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:09 2026 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1ca8b7ab-8952-4355-92a0-9028c13306a9 2026-02-20T22:38:10.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:10.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:10.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_14 2026-02-20T22:38:10.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:10.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:10.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:38:10.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:10.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:38:10.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:10.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:10.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 51908608 bytes 2026-02-20T22:38:10.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 111660032 bytes 2026-02-20T22:38:10.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 166582272 bytes 2026-02-20T22:38:10.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 186323968 bytes 2026-02-20T22:38:11.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 218744832 bytes 2026-02-20T22:38:11.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 240045056 bytes 2026-02-20T22:38:11.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 271610880 bytes 2026-02-20T22:38:11.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 285475840 bytes 2026-02-20T22:38:11.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 319900672 bytes 2026-02-20T22:38:11.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 347233280 bytes 2026-02-20T22:38:11.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 379016192 bytes 2026-02-20T22:38:11.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 425697280 bytes 2026-02-20T22:38:11.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 505462784 bytes 2026-02-20T22:38:11.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 570053632 bytes 2026-02-20T22:38:11.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 635829248 bytes 2026-02-20T22:38:11.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 739023872 bytes 2026-02-20T22:38:11.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 812830720 bytes 2026-02-20T22:38:11.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 861583360 bytes 2026-02-20T22:38:11.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 931258368 bytes 2026-02-20T22:38:11.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 932485120 bytes 2026-02-20T22:38:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 978691072 bytes 2026-02-20T22:38:12.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 998651904 bytes 2026-02-20T22:38:12.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1063428096 bytes 2026-02-20T22:38:12.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1079882752 bytes 2026-02-20T22:38:12.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1102327808 bytes 2026-02-20T22:38:12.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1146643456 bytes 2026-02-20T22:38:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1147254784 bytes 2026-02-20T22:38:12.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1178778624 bytes 2026-02-20T22:38:12.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1228939264 bytes 2026-02-20T22:38:12.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1271297024 bytes 2026-02-20T22:38:12.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1316620288 bytes 2026-02-20T22:38:12.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1320518656 bytes 2026-02-20T22:38:12.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1382978560 bytes 2026-02-20T22:38:12.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1474213888 bytes 2026-02-20T22:38:12.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1526368256 bytes 2026-02-20T22:38:13.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1547352064 bytes 2026-02-20T22:38:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1628421120 bytes 2026-02-20T22:38:13.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1720754176 bytes 2026-02-20T22:38:13.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1747433472 bytes 2026-02-20T22:38:13.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1803916288 bytes 2026-02-20T22:38:13.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1836689408 bytes 2026-02-20T22:38:13.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1870556160 bytes 2026-02-20T22:38:13.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1924808704 bytes 2026-02-20T22:38:13.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1946734592 bytes 2026-02-20T22:38:13.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1951179776 bytes 2026-02-20T22:38:13.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1966731264 bytes 2026-02-20T22:38:13.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2040506368 bytes 2026-02-20T22:38:13.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2117868544 bytes 2026-02-20T22:38:13.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2141831168 bytes 2026-02-20T22:38:13.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2157820928 bytes 2026-02-20T22:38:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2206584832 bytes 2026-02-20T22:38:13.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2227207168 bytes 2026-02-20T22:38:13.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2282756096 bytes 2026-02-20T22:38:14.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2330488832 bytes 2026-02-20T22:38:14.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2428391424 bytes 2026-02-20T22:38:14.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2465472512 bytes 2026-02-20T22:38:14.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2476984320 bytes 2026-02-20T22:38:14.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2514092032 bytes 2026-02-20T22:38:14.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2544741376 bytes 2026-02-20T22:38:14.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2566740992 bytes 2026-02-20T22:38:14.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2594341888 bytes 2026-02-20T22:38:14.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2647516160 bytes 2026-02-20T22:38:14.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2662511616 bytes 2026-02-20T22:38:14.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2702184448 bytes 2026-02-20T22:38:14.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2723345408 bytes 2026-02-20T22:38:14.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2736155648 bytes 2026-02-20T22:38:14.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2736739328 bytes 2026-02-20T22:38:14.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2799355904 bytes 2026-02-20T22:38:14.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2867588096 bytes 2026-02-20T22:38:14.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2938990592 bytes 2026-02-20T22:38:15.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3004985344 bytes 2026-02-20T22:38:15.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3096473600 bytes 2026-02-20T22:38:15.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3138513920 bytes 2026-02-20T22:38:15.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3173798912 bytes 2026-02-20T22:38:15.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3197502464 bytes 2026-02-20T22:38:15.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3204963328 bytes 2026-02-20T22:38:15.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3219788800 bytes 2026-02-20T22:38:15.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3243926528 bytes 2026-02-20T22:38:15.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3264359424 bytes 2026-02-20T22:38:15.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3319489536 bytes 2026-02-20T22:38:15.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3350814720 bytes 2026-02-20T22:38:15.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3382494208 bytes 2026-02-20T22:38:15.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3402535936 bytes 2026-02-20T22:38:15.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3439452160 bytes 2026-02-20T22:38:15.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3456393216 bytes 2026-02-20T22:38:15.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3463792640 bytes 2026-02-20T22:38:15.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3533258752 bytes 2026-02-20T22:38:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3589359616 bytes 2026-02-20T22:38:15.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3607054336 bytes 2026-02-20T22:38:16.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3671666688 bytes 2026-02-20T22:38:16.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3707599872 bytes 2026-02-20T22:38:16.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3718105088 bytes 2026-02-20T22:38:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3743618048 bytes 2026-02-20T22:38:16.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3784361984 bytes 2026-02-20T22:38:16.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3839815680 bytes 2026-02-20T22:38:16.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3904651264 bytes 2026-02-20T22:38:16.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3937068032 bytes 2026-02-20T22:38:16.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3951780864 bytes 2026-02-20T22:38:16.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3998445568 bytes 2026-02-20T22:38:16.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4050145280 bytes 2026-02-20T22:38:16.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4102039552 bytes 2026-02-20T22:38:16.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4130333696 bytes 2026-02-20T22:38:16.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4186249216 bytes 2026-02-20T22:38:16.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4273284096 bytes 2026-02-20T22:38:16.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4334694400 bytes 2026-02-20T22:38:16.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4399086592 bytes 2026-02-20T22:38:17.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4447292416 bytes 2026-02-20T22:38:17.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4482933760 bytes 2026-02-20T22:38:17.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4489442304 bytes 2026-02-20T22:38:17.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4518276096 bytes 2026-02-20T22:38:17.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4545329152 bytes 2026-02-20T22:38:17.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4553251840 bytes 2026-02-20T22:38:17.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4591187968 bytes 2026-02-20T22:38:17.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4620385280 bytes 2026-02-20T22:38:17.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4660954112 bytes 2026-02-20T22:38:17.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4730125312 bytes 2026-02-20T22:38:17.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4767767552 bytes 2026-02-20T22:38:17.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4815302656 bytes 2026-02-20T22:38:17.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4878085120 bytes 2026-02-20T22:38:17.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4951382016 bytes 2026-02-20T22:38:17.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5042639872 bytes 2026-02-20T22:38:17.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5118642176 bytes 2026-02-20T22:38:17.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5135932416 bytes 2026-02-20T22:38:18.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5179764736 bytes 2026-02-20T22:38:18.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5204855808 bytes 2026-02-20T22:38:18.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5268896768 bytes 2026-02-20T22:38:18.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5300114432 bytes 2026-02-20T22:38:18.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5320614912 bytes 2026-02-20T22:38:18.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5351013376 bytes 2026-02-20T22:38:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5375614976 bytes 2026-02-20T22:38:18.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5459598336 bytes 2026-02-20T22:38:18.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5506762752 bytes 2026-02-20T22:38:18.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5540355072 bytes 2026-02-20T22:38:18.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5631859712 bytes 2026-02-20T22:38:18.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5651955712 bytes 2026-02-20T22:38:18.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5667646464 bytes 2026-02-20T22:38:18.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5707406336 bytes 2026-02-20T22:38:18.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5788393472 bytes 2026-02-20T22:38:18.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5825628160 bytes 2026-02-20T22:38:18.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5872629760 bytes 2026-02-20T22:38:18.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5928109056 bytes 2026-02-20T22:38:19.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5994678272 bytes 2026-02-20T22:38:19.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6048653312 bytes 2026-02-20T22:38:19.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6077432832 bytes 2026-02-20T22:38:19.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6113997824 bytes 2026-02-20T22:38:19.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6137461760 bytes 2026-02-20T22:38:19.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6187865088 bytes 2026-02-20T22:38:19.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6200840192 bytes 2026-02-20T22:38:19.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6272834560 bytes 2026-02-20T22:38:19.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6291709952 bytes 2026-02-20T22:38:19.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6349533184 bytes 2026-02-20T22:38:19.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6421116928 bytes 2026-02-20T22:38:19.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6426715136 bytes 2026-02-20T22:38:19.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6454712320 bytes 2026-02-20T22:38:19.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6510973952 bytes 2026-02-20T22:38:19.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6528792576 bytes 2026-02-20T22:38:19.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6539162624 bytes 2026-02-20T22:38:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6567059456 bytes 2026-02-20T22:38:20.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6620771328 bytes 2026-02-20T22:38:20.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6655514624 bytes 2026-02-20T22:38:20.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6722606080 bytes 2026-02-20T22:38:20.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6738609152 bytes 2026-02-20T22:38:20.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6785983488 bytes 2026-02-20T22:38:20.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6829015040 bytes 2026-02-20T22:38:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6884334592 bytes 2026-02-20T22:38:20.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6987552768 bytes 2026-02-20T22:38:20.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7081390080 bytes 2026-02-20T22:38:20.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7129385984 bytes 2026-02-20T22:38:20.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7158310912 bytes 2026-02-20T22:38:20.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7310074880 bytes 2026-02-20T22:38:20.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7371715584 bytes 2026-02-20T22:38:20.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7395136512 bytes 2026-02-20T22:38:20.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7451602944 bytes 2026-02-20T22:38:20.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7474369536 bytes 2026-02-20T22:38:21.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7510370304 bytes 2026-02-20T22:38:21.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7533661184 bytes 2026-02-20T22:38:21.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7576677376 bytes 2026-02-20T22:38:21.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7633283072 bytes 2026-02-20T22:38:21.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7666397184 bytes 2026-02-20T22:38:21.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7747341312 bytes 2026-02-20T22:38:21.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7788122112 bytes 2026-02-20T22:38:21.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7818772480 bytes 2026-02-20T22:38:21.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7854008320 bytes 2026-02-20T22:38:21.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7877104640 bytes 2026-02-20T22:38:21.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7921508352 bytes 2026-02-20T22:38:21.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7951541248 bytes 2026-02-20T22:38:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8005926912 bytes 2026-02-20T22:38:21.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8101497856 bytes 2026-02-20T22:38:21.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8210753536 bytes 2026-02-20T22:38:21.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8312790016 bytes 2026-02-20T22:38:22.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8363883520 bytes 2026-02-20T22:38:22.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8410044416 bytes 2026-02-20T22:38:22.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8410653696 bytes 2026-02-20T22:38:22.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8466927616 bytes 2026-02-20T22:38:22.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8531229696 bytes 2026-02-20T22:38:22.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8536927232 bytes 2026-02-20T22:38:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8620925952 bytes 2026-02-20T22:38:22.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8664798208 bytes 2026-02-20T22:38:22.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8716956672 bytes 2026-02-20T22:38:22.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8724435968 bytes 2026-02-20T22:38:22.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8775230464 bytes 2026-02-20T22:38:22.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8839678976 bytes 2026-02-20T22:38:22.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8887641088 bytes 2026-02-20T22:38:22.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8964563968 bytes 2026-02-20T22:38:22.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9005541376 bytes 2026-02-20T22:38:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9040065536 bytes 2026-02-20T22:38:22.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9077728256 bytes 2026-02-20T22:38:23.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9118820352 bytes 2026-02-20T22:38:23.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9179603968 bytes 2026-02-20T22:38:23.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9233359872 bytes 2026-02-20T22:38:23.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9251779584 bytes 2026-02-20T22:38:23.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9323798528 bytes 2026-02-20T22:38:23.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9365949440 bytes 2026-02-20T22:38:23.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9383237632 bytes 2026-02-20T22:38:23.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9406609408 bytes 2026-02-20T22:38:23.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9459658752 bytes 2026-02-20T22:38:23.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9495197696 bytes 2026-02-20T22:38:23.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9570004992 bytes 2026-02-20T22:38:23.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9623776256 bytes 2026-02-20T22:38:23.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9692492800 bytes 2026-02-20T22:38:23.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9693134848 bytes 2026-02-20T22:38:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9760368640 bytes 2026-02-20T22:38:23.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9788790784 bytes 2026-02-20T22:38:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9799422976 bytes 2026-02-20T22:38:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9804367872 bytes 2026-02-20T22:38:24.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9893351424 bytes 2026-02-20T22:38:24.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 9931730944 bytes 2026-02-20T22:38:24.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 9943910400 bytes 2026-02-20T22:38:24.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 10018114560 bytes 2026-02-20T22:38:24.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 10046051328 bytes 2026-02-20T22:38:24.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 10150140928 bytes 2026-02-20T22:38:24.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23200 IOs, 10185216000 bytes 2026-02-20T22:38:24.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23300 IOs, 10206950400 bytes 2026-02-20T22:38:24.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23400 IOs, 10234823680 bytes 2026-02-20T22:38:24.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23500 IOs, 10284722176 bytes 2026-02-20T22:38:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_15 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_15 128M 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:24.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:24.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:24.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:38:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:24.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2026-02-20T22:38:25.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:25.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2026-02-20T22:38:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_15': 2026-02-20T22:38:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:25.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1308bdcb2139 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1308bdcb2139 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:24 2026 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:24 2026 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:24 2026 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7d4c2418-4884-49bd-8deb-da886a79291a 2026-02-20T22:38:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_15 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:38:25.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:25.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:38:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:25.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 60623872 bytes 2026-02-20T22:38:25.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 98112512 bytes 2026-02-20T22:38:25.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 128882688 bytes 2026-02-20T22:38:25.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 141049856 bytes 2026-02-20T22:38:25.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 178778112 bytes 2026-02-20T22:38:26.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 221236224 bytes 2026-02-20T22:38:26.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 295555072 bytes 2026-02-20T22:38:26.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 326010880 bytes 2026-02-20T22:38:26.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 364833792 bytes 2026-02-20T22:38:26.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 385232896 bytes 2026-02-20T22:38:26.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 394372096 bytes 2026-02-20T22:38:26.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 466155520 bytes 2026-02-20T22:38:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 479442944 bytes 2026-02-20T22:38:26.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 535959552 bytes 2026-02-20T22:38:26.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 564652032 bytes 2026-02-20T22:38:26.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 599782400 bytes 2026-02-20T22:38:26.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 650757120 bytes 2026-02-20T22:38:26.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 689738752 bytes 2026-02-20T22:38:26.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 709197824 bytes 2026-02-20T22:38:26.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 723718144 bytes 2026-02-20T22:38:26.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 777345024 bytes 2026-02-20T22:38:26.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 783870976 bytes 2026-02-20T22:38:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 844667904 bytes 2026-02-20T22:38:27.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 876377088 bytes 2026-02-20T22:38:27.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 931408896 bytes 2026-02-20T22:38:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 975120384 bytes 2026-02-20T22:38:27.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 999189504 bytes 2026-02-20T22:38:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1073103872 bytes 2026-02-20T22:38:27.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1101630464 bytes 2026-02-20T22:38:27.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1139488768 bytes 2026-02-20T22:38:27.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1216290816 bytes 2026-02-20T22:38:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1295918080 bytes 2026-02-20T22:38:27.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1357496320 bytes 2026-02-20T22:38:27.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1405238272 bytes 2026-02-20T22:38:27.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1492993024 bytes 2026-02-20T22:38:27.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1493783552 bytes 2026-02-20T22:38:27.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1516817408 bytes 2026-02-20T22:38:27.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1584083968 bytes 2026-02-20T22:38:28.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1587979264 bytes 2026-02-20T22:38:28.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1621900288 bytes 2026-02-20T22:38:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1622499328 bytes 2026-02-20T22:38:28.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1635778560 bytes 2026-02-20T22:38:28.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1681608704 bytes 2026-02-20T22:38:28.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1742520320 bytes 2026-02-20T22:38:28.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1802544128 bytes 2026-02-20T22:38:28.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1852417024 bytes 2026-02-20T22:38:28.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1916173312 bytes 2026-02-20T22:38:28.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1969897472 bytes 2026-02-20T22:38:28.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1970507776 bytes 2026-02-20T22:38:28.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2001468416 bytes 2026-02-20T22:38:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2113813504 bytes 2026-02-20T22:38:28.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2157803520 bytes 2026-02-20T22:38:28.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2259898368 bytes 2026-02-20T22:38:28.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2310192128 bytes 2026-02-20T22:38:29.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2331730944 bytes 2026-02-20T22:38:29.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2381684736 bytes 2026-02-20T22:38:29.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2382301184 bytes 2026-02-20T22:38:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2388763648 bytes 2026-02-20T22:38:29.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2389378048 bytes 2026-02-20T22:38:29.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2460860416 bytes 2026-02-20T22:38:29.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2508167168 bytes 2026-02-20T22:38:29.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2556535808 bytes 2026-02-20T22:38:29.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2557149184 bytes 2026-02-20T22:38:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2577381376 bytes 2026-02-20T22:38:29.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2594197504 bytes 2026-02-20T22:38:29.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2605053952 bytes 2026-02-20T22:38:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2646696960 bytes 2026-02-20T22:38:29.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2678344704 bytes 2026-02-20T22:38:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2728053760 bytes 2026-02-20T22:38:29.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2772549632 bytes 2026-02-20T22:38:29.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2802318336 bytes 2026-02-20T22:38:29.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2818352128 bytes 2026-02-20T22:38:29.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2887485440 bytes 2026-02-20T22:38:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2930705408 bytes 2026-02-20T22:38:30.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2948746240 bytes 2026-02-20T22:38:30.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2976657408 bytes 2026-02-20T22:38:30.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2995286016 bytes 2026-02-20T22:38:30.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3025211392 bytes 2026-02-20T22:38:30.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3066018816 bytes 2026-02-20T22:38:30.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3121281024 bytes 2026-02-20T22:38:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3176557568 bytes 2026-02-20T22:38:30.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3228188672 bytes 2026-02-20T22:38:30.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3304865792 bytes 2026-02-20T22:38:30.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3306331136 bytes 2026-02-20T22:38:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:30.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_16 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:38:30.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_16 128M 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:30.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:30.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:30.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:30.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2026-02-20T22:38:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:30.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:30.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2026-02-20T22:38:31.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:31.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_16': 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13149ccff7a4 2026-02-20T22:38:31.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13149ccff7a4 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:30 2026 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:30 2026 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:30 2026 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: dd77e808-55a0-4c1a-a3d7-263a1ff1cd0d 2026-02-20T22:38:31.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_16 2026-02-20T22:38:31.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:38:31.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=8 2026-02-20T22:38:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:31.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 31256576 bytes 2026-02-20T22:38:31.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 57720832 bytes 2026-02-20T22:38:31.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 106858496 bytes 2026-02-20T22:38:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 135953408 bytes 2026-02-20T22:38:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 148362240 bytes 2026-02-20T22:38:32.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 168771584 bytes 2026-02-20T22:38:32.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 254730240 bytes 2026-02-20T22:38:32.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 294269952 bytes 2026-02-20T22:38:32.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 320196608 bytes 2026-02-20T22:38:32.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 332628992 bytes 2026-02-20T22:38:32.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 415422464 bytes 2026-02-20T22:38:32.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 416060416 bytes 2026-02-20T22:38:32.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 473185280 bytes 2026-02-20T22:38:32.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 534522880 bytes 2026-02-20T22:38:32.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 573484032 bytes 2026-02-20T22:38:32.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 615658496 bytes 2026-02-20T22:38:32.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 704148480 bytes 2026-02-20T22:38:32.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 739048448 bytes 2026-02-20T22:38:32.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 763672576 bytes 2026-02-20T22:38:33.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 802515968 bytes 2026-02-20T22:38:33.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 851406848 bytes 2026-02-20T22:38:33.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 887409664 bytes 2026-02-20T22:38:33.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 896993280 bytes 2026-02-20T22:38:33.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 942962688 bytes 2026-02-20T22:38:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 970001408 bytes 2026-02-20T22:38:33.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1027728384 bytes 2026-02-20T22:38:33.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1074197504 bytes 2026-02-20T22:38:33.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1151208448 bytes 2026-02-20T22:38:33.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1174111232 bytes 2026-02-20T22:38:33.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1196954624 bytes 2026-02-20T22:38:33.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1245534208 bytes 2026-02-20T22:38:33.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1287094272 bytes 2026-02-20T22:38:33.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1349831680 bytes 2026-02-20T22:38:33.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1350433792 bytes 2026-02-20T22:38:33.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1385627648 bytes 2026-02-20T22:38:33.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1424508928 bytes 2026-02-20T22:38:33.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1448031232 bytes 2026-02-20T22:38:34.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1504509952 bytes 2026-02-20T22:38:34.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1516374016 bytes 2026-02-20T22:38:34.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1555420160 bytes 2026-02-20T22:38:34.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1587353600 bytes 2026-02-20T22:38:34.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1649107968 bytes 2026-02-20T22:38:34.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1710428160 bytes 2026-02-20T22:38:34.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1785673728 bytes 2026-02-20T22:38:34.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1859751936 bytes 2026-02-20T22:38:34.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1931456512 bytes 2026-02-20T22:38:34.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1943317504 bytes 2026-02-20T22:38:34.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1947561984 bytes 2026-02-20T22:38:34.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1972242432 bytes 2026-02-20T22:38:34.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1994563584 bytes 2026-02-20T22:38:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2019546112 bytes 2026-02-20T22:38:34.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2063531008 bytes 2026-02-20T22:38:34.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2126251008 bytes 2026-02-20T22:38:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2153406464 bytes 2026-02-20T22:38:34.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2166592512 bytes 2026-02-20T22:38:35.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2267866112 bytes 2026-02-20T22:38:35.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2316157952 bytes 2026-02-20T22:38:35.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2347488256 bytes 2026-02-20T22:38:35.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2389182464 bytes 2026-02-20T22:38:35.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2461202432 bytes 2026-02-20T22:38:35.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2602344448 bytes 2026-02-20T22:38:35.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2619287552 bytes 2026-02-20T22:38:35.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2711974912 bytes 2026-02-20T22:38:35.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2712619008 bytes 2026-02-20T22:38:35.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2720993280 bytes 2026-02-20T22:38:35.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2755901440 bytes 2026-02-20T22:38:35.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2818946048 bytes 2026-02-20T22:38:35.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2873860096 bytes 2026-02-20T22:38:35.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2919946240 bytes 2026-02-20T22:38:35.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3031126016 bytes 2026-02-20T22:38:36.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3096563712 bytes 2026-02-20T22:38:36.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3139324928 bytes 2026-02-20T22:38:36.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3184114688 bytes 2026-02-20T22:38:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3201620992 bytes 2026-02-20T22:38:36.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3221324800 bytes 2026-02-20T22:38:36.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3259264000 bytes 2026-02-20T22:38:36.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3287274496 bytes 2026-02-20T22:38:36.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3311799296 bytes 2026-02-20T22:38:36.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3341245440 bytes 2026-02-20T22:38:36.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3358946304 bytes 2026-02-20T22:38:36.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3428593664 bytes 2026-02-20T22:38:36.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3485360128 bytes 2026-02-20T22:38:36.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3512914944 bytes 2026-02-20T22:38:36.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3584494592 bytes 2026-02-20T22:38:36.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3643224064 bytes 2026-02-20T22:38:36.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3655412736 bytes 2026-02-20T22:38:36.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3680521216 bytes 2026-02-20T22:38:36.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3701097472 bytes 2026-02-20T22:38:37.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3771819008 bytes 2026-02-20T22:38:37.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3778800640 bytes 2026-02-20T22:38:37.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3811864576 bytes 2026-02-20T22:38:37.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3891758080 bytes 2026-02-20T22:38:37.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3923411968 bytes 2026-02-20T22:38:37.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3944032256 bytes 2026-02-20T22:38:37.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3995486208 bytes 2026-02-20T22:38:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4081094656 bytes 2026-02-20T22:38:37.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4106510336 bytes 2026-02-20T22:38:37.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4130040832 bytes 2026-02-20T22:38:37.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4183802880 bytes 2026-02-20T22:38:37.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4214850560 bytes 2026-02-20T22:38:37.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4224494592 bytes 2026-02-20T22:38:37.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4238828544 bytes 2026-02-20T22:38:37.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4316158976 bytes 2026-02-20T22:38:37.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4348041216 bytes 2026-02-20T22:38:37.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4372836352 bytes 2026-02-20T22:38:37.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4408579072 bytes 2026-02-20T22:38:37.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4474509312 bytes 2026-02-20T22:38:38.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4556249088 bytes 2026-02-20T22:38:38.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4632514560 bytes 2026-02-20T22:38:38.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4696282112 bytes 2026-02-20T22:38:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4791892992 bytes 2026-02-20T22:38:38.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4822068224 bytes 2026-02-20T22:38:38.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4881793024 bytes 2026-02-20T22:38:38.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4882419712 bytes 2026-02-20T22:38:38.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4978618368 bytes 2026-02-20T22:38:38.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5033105408 bytes 2026-02-20T22:38:38.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5110300672 bytes 2026-02-20T22:38:38.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5133880320 bytes 2026-02-20T22:38:38.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5159011328 bytes 2026-02-20T22:38:38.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5163578368 bytes 2026-02-20T22:38:38.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5199376384 bytes 2026-02-20T22:38:38.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5226565632 bytes 2026-02-20T22:38:38.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5297461248 bytes 2026-02-20T22:38:39.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5326389248 bytes 2026-02-20T22:38:39.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5349254144 bytes 2026-02-20T22:38:39.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5401899008 bytes 2026-02-20T22:38:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5454415872 bytes 2026-02-20T22:38:39.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5493280768 bytes 2026-02-20T22:38:39.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5588889600 bytes 2026-02-20T22:38:39.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5614305280 bytes 2026-02-20T22:38:39.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5644700672 bytes 2026-02-20T22:38:39.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5713232896 bytes 2026-02-20T22:38:39.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5756944384 bytes 2026-02-20T22:38:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5780861952 bytes 2026-02-20T22:38:39.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5798918144 bytes 2026-02-20T22:38:39.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5820147712 bytes 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_17 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2026-02-20T22:38:39.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_17 128M 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:39.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:39.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2026-02-20T22:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:39.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2026-02-20T22:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:39.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2026-02-20T22:38:40.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:40.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_17': 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1320c505063 2026-02-20T22:38:40.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1320c505063 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:39 2026 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:39 2026 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:39 2026 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 58103425-2768-48ad-acca-e3c306a52862 2026-02-20T22:38:40.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_17 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:38:40.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:38:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:40.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:40.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 79334400 bytes 2026-02-20T22:38:40.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 133844992 bytes 2026-02-20T22:38:40.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 144998400 bytes 2026-02-20T22:38:40.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 193983488 bytes 2026-02-20T22:38:41.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 219344896 bytes 2026-02-20T22:38:41.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 263025664 bytes 2026-02-20T22:38:41.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 265086976 bytes 2026-02-20T22:38:41.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 319010816 bytes 2026-02-20T22:38:41.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 322825216 bytes 2026-02-20T22:38:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 365768704 bytes 2026-02-20T22:38:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 385381376 bytes 2026-02-20T22:38:41.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 406203392 bytes 2026-02-20T22:38:41.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 486496256 bytes 2026-02-20T22:38:41.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 514982912 bytes 2026-02-20T22:38:41.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 515607552 bytes 2026-02-20T22:38:41.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 588660736 bytes 2026-02-20T22:38:41.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 589257728 bytes 2026-02-20T22:38:41.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 648904704 bytes 2026-02-20T22:38:41.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 666313728 bytes 2026-02-20T22:38:41.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 725090304 bytes 2026-02-20T22:38:41.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 747778048 bytes 2026-02-20T22:38:41.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 806336512 bytes 2026-02-20T22:38:42.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 876906496 bytes 2026-02-20T22:38:42.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 907516928 bytes 2026-02-20T22:38:42.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 966710272 bytes 2026-02-20T22:38:42.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1033435136 bytes 2026-02-20T22:38:42.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1074060288 bytes 2026-02-20T22:38:42.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1134556160 bytes 2026-02-20T22:38:42.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1174252544 bytes 2026-02-20T22:38:42.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1241673728 bytes 2026-02-20T22:38:42.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1261416448 bytes 2026-02-20T22:38:42.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1327684608 bytes 2026-02-20T22:38:42.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1399584768 bytes 2026-02-20T22:38:42.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1481013248 bytes 2026-02-20T22:38:42.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1492699136 bytes 2026-02-20T22:38:42.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1521131520 bytes 2026-02-20T22:38:42.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1544574976 bytes 2026-02-20T22:38:43.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1600971776 bytes 2026-02-20T22:38:43.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1657110528 bytes 2026-02-20T22:38:43.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1746557952 bytes 2026-02-20T22:38:43.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1801675776 bytes 2026-02-20T22:38:43.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1815075840 bytes 2026-02-20T22:38:43.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1862456320 bytes 2026-02-20T22:38:43.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1914722304 bytes 2026-02-20T22:38:43.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1929549824 bytes 2026-02-20T22:38:43.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1968909312 bytes 2026-02-20T22:38:43.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2065377280 bytes 2026-02-20T22:38:43.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2099994624 bytes 2026-02-20T22:38:43.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2162117632 bytes 2026-02-20T22:38:43.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2195678208 bytes 2026-02-20T22:38:43.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2268626944 bytes 2026-02-20T22:38:43.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2295229440 bytes 2026-02-20T22:38:43.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2309609472 bytes 2026-02-20T22:38:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2311897088 bytes 2026-02-20T22:38:44.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2385922048 bytes 2026-02-20T22:38:44.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2418820096 bytes 2026-02-20T22:38:44.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2472326144 bytes 2026-02-20T22:38:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2472945664 bytes 2026-02-20T22:38:44.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2486764544 bytes 2026-02-20T22:38:44.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2548323328 bytes 2026-02-20T22:38:44.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2680375296 bytes 2026-02-20T22:38:44.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2699918336 bytes 2026-02-20T22:38:44.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2779595776 bytes 2026-02-20T22:38:44.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2857304064 bytes 2026-02-20T22:38:44.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2903827456 bytes 2026-02-20T22:38:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2976319488 bytes 2026-02-20T22:38:44.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3013774336 bytes 2026-02-20T22:38:44.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3053970432 bytes 2026-02-20T22:38:44.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3090822144 bytes 2026-02-20T22:38:44.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3110258688 bytes 2026-02-20T22:38:45.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3139574784 bytes 2026-02-20T22:38:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3169047552 bytes 2026-02-20T22:38:45.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3215050752 bytes 2026-02-20T22:38:45.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3254703104 bytes 2026-02-20T22:38:45.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3281609728 bytes 2026-02-20T22:38:45.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3361235968 bytes 2026-02-20T22:38:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3452539904 bytes 2026-02-20T22:38:45.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3485931520 bytes 2026-02-20T22:38:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3543999488 bytes 2026-02-20T22:38:45.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3544612864 bytes 2026-02-20T22:38:45.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3603946496 bytes 2026-02-20T22:38:45.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3719678976 bytes 2026-02-20T22:38:45.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3830733824 bytes 2026-02-20T22:38:45.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3855994880 bytes 2026-02-20T22:38:45.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3886170112 bytes 2026-02-20T22:38:46.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3897433088 bytes 2026-02-20T22:38:46.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3959333888 bytes 2026-02-20T22:38:46.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3984753664 bytes 2026-02-20T22:38:46.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4046921728 bytes 2026-02-20T22:38:46.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4055774208 bytes 2026-02-20T22:38:46.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4104409088 bytes 2026-02-20T22:38:46.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4133668864 bytes 2026-02-20T22:38:46.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4154737664 bytes 2026-02-20T22:38:46.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4185588736 bytes 2026-02-20T22:38:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4212159488 bytes 2026-02-20T22:38:46.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4249536512 bytes 2026-02-20T22:38:46.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4256087040 bytes 2026-02-20T22:38:46.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4286146560 bytes 2026-02-20T22:38:46.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4333119488 bytes 2026-02-20T22:38:46.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4405672960 bytes 2026-02-20T22:38:46.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4422910976 bytes 2026-02-20T22:38:46.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4434873344 bytes 2026-02-20T22:38:46.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4458125312 bytes 2026-02-20T22:38:46.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4474870784 bytes 2026-02-20T22:38:47.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4550888448 bytes 2026-02-20T22:38:47.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4559771648 bytes 2026-02-20T22:38:47.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4591540224 bytes 2026-02-20T22:38:47.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4625993728 bytes 2026-02-20T22:38:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4645009408 bytes 2026-02-20T22:38:47.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4694970368 bytes 2026-02-20T22:38:47.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4757710848 bytes 2026-02-20T22:38:47.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4769607680 bytes 2026-02-20T22:38:47.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4773864448 bytes 2026-02-20T22:38:47.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4797941760 bytes 2026-02-20T22:38:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4801203200 bytes 2026-02-20T22:38:47.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4818433024 bytes 2026-02-20T22:38:47.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4884608000 bytes 2026-02-20T22:38:47.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4896966656 bytes 2026-02-20T22:38:47.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4902447104 bytes 2026-02-20T22:38:47.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4912917504 bytes 2026-02-20T22:38:47.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4979020800 bytes 2026-02-20T22:38:47.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4984578048 bytes 2026-02-20T22:38:47.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5002450944 bytes 2026-02-20T22:38:47.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5069126656 bytes 2026-02-20T22:38:47.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5069724672 bytes 2026-02-20T22:38:48.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5110725632 bytes 2026-02-20T22:38:48.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5182774272 bytes 2026-02-20T22:38:48.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5227673600 bytes 2026-02-20T22:38:48.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5287124992 bytes 2026-02-20T22:38:48.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5316842496 bytes 2026-02-20T22:38:48.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5348670464 bytes 2026-02-20T22:38:48.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5403311104 bytes 2026-02-20T22:38:48.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5416159232 bytes 2026-02-20T22:38:48.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5474264064 bytes 2026-02-20T22:38:48.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5502467072 bytes 2026-02-20T22:38:48.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5519431680 bytes 2026-02-20T22:38:48.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5574499328 bytes 2026-02-20T22:38:48.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5658959872 bytes 2026-02-20T22:38:48.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5720470528 bytes 2026-02-20T22:38:48.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5738319872 bytes 2026-02-20T22:38:48.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5787833344 bytes 2026-02-20T22:38:48.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5788431360 bytes 2026-02-20T22:38:49.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5865156608 bytes 2026-02-20T22:38:49.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5872943104 bytes 2026-02-20T22:38:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5896450048 bytes 2026-02-20T22:38:49.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5956544512 bytes 2026-02-20T22:38:49.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6009096192 bytes 2026-02-20T22:38:49.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6049991680 bytes 2026-02-20T22:38:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6074437632 bytes 2026-02-20T22:38:49.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6179333120 bytes 2026-02-20T22:38:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6234357760 bytes 2026-02-20T22:38:49.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6279775232 bytes 2026-02-20T22:38:49.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6309123072 bytes 2026-02-20T22:38:49.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6368430080 bytes 2026-02-20T22:38:49.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6412703744 bytes 2026-02-20T22:38:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6438564864 bytes 2026-02-20T22:38:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6509409280 bytes 2026-02-20T22:38:49.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6527130624 bytes 2026-02-20T22:38:50.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6575033344 bytes 2026-02-20T22:38:50.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6600756224 bytes 2026-02-20T22:38:50.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6657093632 bytes 2026-02-20T22:38:50.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6712134656 bytes 2026-02-20T22:38:50.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6736400384 bytes 2026-02-20T22:38:50.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6770056192 bytes 2026-02-20T22:38:50.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6790555648 bytes 2026-02-20T22:38:50.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6851574784 bytes 2026-02-20T22:38:50.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6877873152 bytes 2026-02-20T22:38:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6920240128 bytes 2026-02-20T22:38:50.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7019032576 bytes 2026-02-20T22:38:50.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_18 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_18 128M 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:50.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:50.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2026-02-20T22:38:51.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:51.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_18': 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 132c77c0c7c4 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.132c77c0c7c4 2026-02-20T22:38:51.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:50 2026 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:50 2026 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:50 2026 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2e7d6018-b8f8-4aa9-a620-9abcba9ad630 2026-02-20T22:38:51.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_18 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:38:51.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:38:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:51.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 68452352 bytes 2026-02-20T22:38:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 69079040 bytes 2026-02-20T22:38:51.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 90864640 bytes 2026-02-20T22:38:51.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 114040832 bytes 2026-02-20T22:38:52.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 153746432 bytes 2026-02-20T22:38:52.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 202199040 bytes 2026-02-20T22:38:52.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 274285568 bytes 2026-02-20T22:38:52.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 317235200 bytes 2026-02-20T22:38:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 317851648 bytes 2026-02-20T22:38:52.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 340761600 bytes 2026-02-20T22:38:52.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 366392320 bytes 2026-02-20T22:38:52.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 384930816 bytes 2026-02-20T22:38:52.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 430631936 bytes 2026-02-20T22:38:52.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 477560832 bytes 2026-02-20T22:38:52.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 513378304 bytes 2026-02-20T22:38:52.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 526984192 bytes 2026-02-20T22:38:52.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 538944512 bytes 2026-02-20T22:38:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 559595520 bytes 2026-02-20T22:38:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 597738496 bytes 2026-02-20T22:38:52.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 624179200 bytes 2026-02-20T22:38:52.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 678408192 bytes 2026-02-20T22:38:52.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 722625536 bytes 2026-02-20T22:38:52.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 802359296 bytes 2026-02-20T22:38:53.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 862994432 bytes 2026-02-20T22:38:53.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 884675584 bytes 2026-02-20T22:38:53.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 899007488 bytes 2026-02-20T22:38:53.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 933836800 bytes 2026-02-20T22:38:53.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 948731904 bytes 2026-02-20T22:38:53.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 996414464 bytes 2026-02-20T22:38:53.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1003618304 bytes 2026-02-20T22:38:53.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1052332032 bytes 2026-02-20T22:38:53.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1083335680 bytes 2026-02-20T22:38:53.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1120365568 bytes 2026-02-20T22:38:53.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1136680960 bytes 2026-02-20T22:38:53.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1153873920 bytes 2026-02-20T22:38:53.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1174554624 bytes 2026-02-20T22:38:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1233182720 bytes 2026-02-20T22:38:53.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1272543232 bytes 2026-02-20T22:38:53.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1273171968 bytes 2026-02-20T22:38:53.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1273775104 bytes 2026-02-20T22:38:53.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1323338752 bytes 2026-02-20T22:38:53.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1329699840 bytes 2026-02-20T22:38:53.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1394117632 bytes 2026-02-20T22:38:54.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1461578752 bytes 2026-02-20T22:38:54.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1547463680 bytes 2026-02-20T22:38:54.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1623931904 bytes 2026-02-20T22:38:54.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1624521728 bytes 2026-02-20T22:38:54.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1657434112 bytes 2026-02-20T22:38:54.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1662346240 bytes 2026-02-20T22:38:54.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1708974080 bytes 2026-02-20T22:38:54.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1733666816 bytes 2026-02-20T22:38:54.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1758372864 bytes 2026-02-20T22:38:54.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1805375488 bytes 2026-02-20T22:38:54.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1861849088 bytes 2026-02-20T22:38:54.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 1927816192 bytes 2026-02-20T22:38:54.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 1994692608 bytes 2026-02-20T22:38:54.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2064559104 bytes 2026-02-20T22:38:54.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2069266432 bytes 2026-02-20T22:38:54.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2130503680 bytes 2026-02-20T22:38:54.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2150933504 bytes 2026-02-20T22:38:54.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2176265216 bytes 2026-02-20T22:38:55.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2247576576 bytes 2026-02-20T22:38:55.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2303777792 bytes 2026-02-20T22:38:55.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2371266560 bytes 2026-02-20T22:38:55.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2433756160 bytes 2026-02-20T22:38:55.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2459208704 bytes 2026-02-20T22:38:55.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2507195392 bytes 2026-02-20T22:38:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2525945856 bytes 2026-02-20T22:38:55.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2601390080 bytes 2026-02-20T22:38:55.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2665502720 bytes 2026-02-20T22:38:55.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2690607104 bytes 2026-02-20T22:38:55.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2746072064 bytes 2026-02-20T22:38:55.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2792016896 bytes 2026-02-20T22:38:55.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2802248704 bytes 2026-02-20T22:38:55.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2807885824 bytes 2026-02-20T22:38:55.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2856469504 bytes 2026-02-20T22:38:55.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2857083904 bytes 2026-02-20T22:38:56.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2965598208 bytes 2026-02-20T22:38:56.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2986414080 bytes 2026-02-20T22:38:56.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 2988871680 bytes 2026-02-20T22:38:56.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3024359424 bytes 2026-02-20T22:38:56.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3067099136 bytes 2026-02-20T22:38:56.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3124822016 bytes 2026-02-20T22:38:56.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3140324352 bytes 2026-02-20T22:38:56.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3149331456 bytes 2026-02-20T22:38:56.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3188843520 bytes 2026-02-20T22:38:56.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3199948800 bytes 2026-02-20T22:38:56.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3235056640 bytes 2026-02-20T22:38:56.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3255907328 bytes 2026-02-20T22:38:56.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3322275840 bytes 2026-02-20T22:38:56.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3405873152 bytes 2026-02-20T22:38:56.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3447656448 bytes 2026-02-20T22:38:56.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3486344192 bytes 2026-02-20T22:38:56.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3533380608 bytes 2026-02-20T22:38:56.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3554514944 bytes 2026-02-20T22:38:56.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3632114688 bytes 2026-02-20T22:38:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3634907136 bytes 2026-02-20T22:38:57.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3689019392 bytes 2026-02-20T22:38:57.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3733246976 bytes 2026-02-20T22:38:57.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3764550656 bytes 2026-02-20T22:38:57.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3779675136 bytes 2026-02-20T22:38:57.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 3819568128 bytes 2026-02-20T22:38:57.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 3824253952 bytes 2026-02-20T22:38:57.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 3836684288 bytes 2026-02-20T22:38:57.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 3921472512 bytes 2026-02-20T22:38:57.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 3994973184 bytes 2026-02-20T22:38:57.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4027846656 bytes 2026-02-20T22:38:57.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4103255040 bytes 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_19 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_19 128M 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:57.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:38:57.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:57.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:38:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:38:57.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2026-02-20T22:38:58.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:38:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2026-02-20T22:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_19': 2026-02-20T22:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:38:58.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1338a6ab905e 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1338a6ab905e 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:57 2026 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:57 2026 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:57 2026 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:38:58.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:38:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3c21132a-e262-46d0-a004-e6e7cbc448c9 2026-02-20T22:38:58.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:38:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:38:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_19 2026-02-20T22:38:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:38:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:38:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:38:58.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:38:58.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:38:58.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:38:58.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:38:58.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 9992192 bytes 2026-02-20T22:38:58.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 89773056 bytes 2026-02-20T22:38:58.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 126232576 bytes 2026-02-20T22:38:59.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 199760896 bytes 2026-02-20T22:38:59.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 249443328 bytes 2026-02-20T22:38:59.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 293851136 bytes 2026-02-20T22:38:59.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 349022208 bytes 2026-02-20T22:38:59.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 387321856 bytes 2026-02-20T22:38:59.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 436422656 bytes 2026-02-20T22:38:59.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 540896256 bytes 2026-02-20T22:38:59.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 574470144 bytes 2026-02-20T22:38:59.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 583942144 bytes 2026-02-20T22:38:59.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 661288960 bytes 2026-02-20T22:38:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 705235968 bytes 2026-02-20T22:38:59.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 742623232 bytes 2026-02-20T22:38:59.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 776982528 bytes 2026-02-20T22:38:59.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 781360128 bytes 2026-02-20T22:38:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 815181824 bytes 2026-02-20T22:38:59.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 857451520 bytes 2026-02-20T22:39:00.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 888236032 bytes 2026-02-20T22:39:00.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 903534592 bytes 2026-02-20T22:39:00.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 909423616 bytes 2026-02-20T22:39:00.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 948173824 bytes 2026-02-20T22:39:00.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 992609280 bytes 2026-02-20T22:39:00.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1016241152 bytes 2026-02-20T22:39:00.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1095673856 bytes 2026-02-20T22:39:00.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1112579072 bytes 2026-02-20T22:39:00.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1151857664 bytes 2026-02-20T22:39:00.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1162140672 bytes 2026-02-20T22:39:00.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1209720832 bytes 2026-02-20T22:39:00.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1264997376 bytes 2026-02-20T22:39:00.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1338140672 bytes 2026-02-20T22:39:00.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1435505664 bytes 2026-02-20T22:39:00.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1481254912 bytes 2026-02-20T22:39:00.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1514399744 bytes 2026-02-20T22:39:01.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1534256128 bytes 2026-02-20T22:39:01.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1591365632 bytes 2026-02-20T22:39:01.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1640354816 bytes 2026-02-20T22:39:01.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1687358464 bytes 2026-02-20T22:39:01.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1738476544 bytes 2026-02-20T22:39:01.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1746724864 bytes 2026-02-20T22:39:01.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1766506496 bytes 2026-02-20T22:39:01.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1803663360 bytes 2026-02-20T22:39:01.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1918095360 bytes 2026-02-20T22:39:01.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1961821184 bytes 2026-02-20T22:39:01.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2028285952 bytes 2026-02-20T22:39:01.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2107448320 bytes 2026-02-20T22:39:01.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2151141376 bytes 2026-02-20T22:39:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2246851584 bytes 2026-02-20T22:39:01.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2302687232 bytes 2026-02-20T22:39:02.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2331735040 bytes 2026-02-20T22:39:02.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2357705728 bytes 2026-02-20T22:39:02.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2368885760 bytes 2026-02-20T22:39:02.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2407647232 bytes 2026-02-20T22:39:02.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2428755968 bytes 2026-02-20T22:39:02.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2474301440 bytes 2026-02-20T22:39:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2523218944 bytes 2026-02-20T22:39:02.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2568665088 bytes 2026-02-20T22:39:02.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2600171520 bytes 2026-02-20T22:39:02.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2683826176 bytes 2026-02-20T22:39:02.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2694411264 bytes 2026-02-20T22:39:02.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2763212800 bytes 2026-02-20T22:39:02.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2801863680 bytes 2026-02-20T22:39:02.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2850384896 bytes 2026-02-20T22:39:02.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2893206528 bytes 2026-02-20T22:39:02.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2991862784 bytes 2026-02-20T22:39:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3127022592 bytes 2026-02-20T22:39:03.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3149066240 bytes 2026-02-20T22:39:03.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3186229248 bytes 2026-02-20T22:39:03.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3216214016 bytes 2026-02-20T22:39:03.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3222095872 bytes 2026-02-20T22:39:03.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3257742336 bytes 2026-02-20T22:39:03.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3326468096 bytes 2026-02-20T22:39:03.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3372034048 bytes 2026-02-20T22:39:03.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3415804928 bytes 2026-02-20T22:39:03.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3459288064 bytes 2026-02-20T22:39:03.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3483096064 bytes 2026-02-20T22:39:03.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3539214336 bytes 2026-02-20T22:39:03.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3621361664 bytes 2026-02-20T22:39:03.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3692987392 bytes 2026-02-20T22:39:03.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3749964800 bytes 2026-02-20T22:39:04.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3787033600 bytes 2026-02-20T22:39:04.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3839991808 bytes 2026-02-20T22:39:04.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3897391104 bytes 2026-02-20T22:39:04.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3939981312 bytes 2026-02-20T22:39:04.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4017990656 bytes 2026-02-20T22:39:04.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4069112832 bytes 2026-02-20T22:39:04.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4135139328 bytes 2026-02-20T22:39:04.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4190666752 bytes 2026-02-20T22:39:04.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4250208256 bytes 2026-02-20T22:39:04.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4255689728 bytes 2026-02-20T22:39:04.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4317165568 bytes 2026-02-20T22:39:04.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4341208064 bytes 2026-02-20T22:39:04.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4380433408 bytes 2026-02-20T22:39:04.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4436810752 bytes 2026-02-20T22:39:04.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4494822400 bytes 2026-02-20T22:39:04.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4528684032 bytes 2026-02-20T22:39:05.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4561101824 bytes 2026-02-20T22:39:05.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4603023360 bytes 2026-02-20T22:39:05.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4630006784 bytes 2026-02-20T22:39:05.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4630617088 bytes 2026-02-20T22:39:05.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4674946048 bytes 2026-02-20T22:39:05.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4691443712 bytes 2026-02-20T22:39:05.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4760445952 bytes 2026-02-20T22:39:05.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4803997696 bytes 2026-02-20T22:39:05.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4857503744 bytes 2026-02-20T22:39:05.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4923309056 bytes 2026-02-20T22:39:05.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4950531072 bytes 2026-02-20T22:39:05.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4978359296 bytes 2026-02-20T22:39:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5041081344 bytes 2026-02-20T22:39:05.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5127700480 bytes 2026-02-20T22:39:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5128304640 bytes 2026-02-20T22:39:06.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5193314304 bytes 2026-02-20T22:39:06.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5222439936 bytes 2026-02-20T22:39:06.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5242511360 bytes 2026-02-20T22:39:06.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5315975168 bytes 2026-02-20T22:39:06.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5316584448 bytes 2026-02-20T22:39:06.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5364818944 bytes 2026-02-20T22:39:06.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5391797248 bytes 2026-02-20T22:39:06.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5453892608 bytes 2026-02-20T22:39:06.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5542897664 bytes 2026-02-20T22:39:06.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5561699328 bytes 2026-02-20T22:39:06.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5592030208 bytes 2026-02-20T22:39:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5659345920 bytes 2026-02-20T22:39:06.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5751928832 bytes 2026-02-20T22:39:06.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5756937216 bytes 2026-02-20T22:39:06.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5818317824 bytes 2026-02-20T22:39:07.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5836863488 bytes 2026-02-20T22:39:07.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5881566208 bytes 2026-02-20T22:39:07.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5895738368 bytes 2026-02-20T22:39:07.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5921006592 bytes 2026-02-20T22:39:07.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5980467200 bytes 2026-02-20T22:39:07.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6027460608 bytes 2026-02-20T22:39:07.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6032819200 bytes 2026-02-20T22:39:07.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6049771520 bytes 2026-02-20T22:39:07.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6075672576 bytes 2026-02-20T22:39:07.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6131082240 bytes 2026-02-20T22:39:07.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6163716096 bytes 2026-02-20T22:39:07.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6189811712 bytes 2026-02-20T22:39:07.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6247047168 bytes 2026-02-20T22:39:07.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6294035456 bytes 2026-02-20T22:39:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6350907392 bytes 2026-02-20T22:39:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6423396352 bytes 2026-02-20T22:39:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6490804224 bytes 2026-02-20T22:39:08.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6535220224 bytes 2026-02-20T22:39:08.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6622722048 bytes 2026-02-20T22:39:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6672479232 bytes 2026-02-20T22:39:08.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6730566656 bytes 2026-02-20T22:39:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6746590208 bytes 2026-02-20T22:39:08.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6753826816 bytes 2026-02-20T22:39:08.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6801934336 bytes 2026-02-20T22:39:08.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6859517952 bytes 2026-02-20T22:39:08.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6891388928 bytes 2026-02-20T22:39:08.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6925477888 bytes 2026-02-20T22:39:08.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7026581504 bytes 2026-02-20T22:39:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7071052800 bytes 2026-02-20T22:39:08.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7124369408 bytes 2026-02-20T22:39:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7148056576 bytes 2026-02-20T22:39:09.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7213349888 bytes 2026-02-20T22:39:09.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7241382912 bytes 2026-02-20T22:39:09.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7275873280 bytes 2026-02-20T22:39:09.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7353563136 bytes 2026-02-20T22:39:09.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7473552384 bytes 2026-02-20T22:39:09.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7519351808 bytes 2026-02-20T22:39:09.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7530744832 bytes 2026-02-20T22:39:09.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7563198464 bytes 2026-02-20T22:39:09.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7627459584 bytes 2026-02-20T22:39:09.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7683185664 bytes 2026-02-20T22:39:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7742406656 bytes 2026-02-20T22:39:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7796225024 bytes 2026-02-20T22:39:09.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7836658688 bytes 2026-02-20T22:39:09.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7849326592 bytes 2026-02-20T22:39:09.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7873481728 bytes 2026-02-20T22:39:10.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7907484672 bytes 2026-02-20T22:39:10.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7932625920 bytes 2026-02-20T22:39:10.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7933252608 bytes 2026-02-20T22:39:10.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7999965184 bytes 2026-02-20T22:39:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 8129733632 bytes 2026-02-20T22:39:10.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 8175745024 bytes 2026-02-20T22:39:10.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 8199585792 bytes 2026-02-20T22:39:10.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8298155008 bytes 2026-02-20T22:39:10.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8350689280 bytes 2026-02-20T22:39:10.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8393869312 bytes 2026-02-20T22:39:10.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8419177472 bytes 2026-02-20T22:39:10.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8497394688 bytes 2026-02-20T22:39:10.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8511089664 bytes 2026-02-20T22:39:10.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8521139200 bytes 2026-02-20T22:39:10.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8529675264 bytes 2026-02-20T22:39:10.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8549786624 bytes 2026-02-20T22:39:10.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8604122112 bytes 2026-02-20T22:39:11.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8604746752 bytes 2026-02-20T22:39:11.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8643872768 bytes 2026-02-20T22:39:11.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8672451584 bytes 2026-02-20T22:39:11.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8719187968 bytes 2026-02-20T22:39:11.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8775085056 bytes 2026-02-20T22:39:11.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8840603648 bytes 2026-02-20T22:39:11.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8877035520 bytes 2026-02-20T22:39:11.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8936235008 bytes 2026-02-20T22:39:11.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 9012174848 bytes 2026-02-20T22:39:11.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 9074149376 bytes 2026-02-20T22:39:11.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 9134002176 bytes 2026-02-20T22:39:11.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 9218409472 bytes 2026-02-20T22:39:11.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 9266463744 bytes 2026-02-20T22:39:11.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9312046080 bytes 2026-02-20T22:39:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9368946688 bytes 2026-02-20T22:39:12.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9405062144 bytes 2026-02-20T22:39:12.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9426553856 bytes 2026-02-20T22:39:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9450051584 bytes 2026-02-20T22:39:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9497481216 bytes 2026-02-20T22:39:12.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9550803968 bytes 2026-02-20T22:39:12.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9581444096 bytes 2026-02-20T22:39:12.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9632961536 bytes 2026-02-20T22:39:12.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9676457984 bytes 2026-02-20T22:39:12.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9763426304 bytes 2026-02-20T22:39:12.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9821194240 bytes 2026-02-20T22:39:12.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9863567360 bytes 2026-02-20T22:39:12.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9881236480 bytes 2026-02-20T22:39:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:39:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:39:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:39:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:12.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_20 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_20 128M 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2026-02-20T22:39:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:39:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:39:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:12.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2026-02-20T22:39:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:39:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:12.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2026-02-20T22:39:13.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:39:13.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_20': 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13448009f6b9 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13448009f6b9 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:12 2026 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:12 2026 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:12 2026 2026-02-20T22:39:13.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:39:13.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:39:13.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 84a30fa8-259c-4c04-88f1-8a4e44ec7139 2026-02-20T22:39:13.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_20 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:39:13.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:39:13.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:39:13.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:39:13.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:39:13.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 12129280 bytes 2026-02-20T22:39:13.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 92547072 bytes 2026-02-20T22:39:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 104768512 bytes 2026-02-20T22:39:14.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 141215744 bytes 2026-02-20T22:39:14.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 227382272 bytes 2026-02-20T22:39:14.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 227965952 bytes 2026-02-20T22:39:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 257275904 bytes 2026-02-20T22:39:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 260996096 bytes 2026-02-20T22:39:14.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 285887488 bytes 2026-02-20T22:39:14.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 360634368 bytes 2026-02-20T22:39:14.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 394936320 bytes 2026-02-20T22:39:14.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 451685376 bytes 2026-02-20T22:39:14.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 486717440 bytes 2026-02-20T22:39:14.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 507433984 bytes 2026-02-20T22:39:14.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 543475712 bytes 2026-02-20T22:39:14.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 544080896 bytes 2026-02-20T22:39:14.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 609654784 bytes 2026-02-20T22:39:14.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 656394240 bytes 2026-02-20T22:39:15.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 731515904 bytes 2026-02-20T22:39:15.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 750262272 bytes 2026-02-20T22:39:15.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 861261824 bytes 2026-02-20T22:39:15.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 945792000 bytes 2026-02-20T22:39:15.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1013074944 bytes 2026-02-20T22:39:15.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1046590464 bytes 2026-02-20T22:39:15.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1065943040 bytes 2026-02-20T22:39:15.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1120854016 bytes 2026-02-20T22:39:15.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1171460096 bytes 2026-02-20T22:39:15.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1237698560 bytes 2026-02-20T22:39:15.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1259071488 bytes 2026-02-20T22:39:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1259692032 bytes 2026-02-20T22:39:15.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1310686208 bytes 2026-02-20T22:39:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1343506432 bytes 2026-02-20T22:39:15.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1414932480 bytes 2026-02-20T22:39:15.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1429265408 bytes 2026-02-20T22:39:16.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1497413632 bytes 2026-02-20T22:39:16.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1512572928 bytes 2026-02-20T22:39:16.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1564359680 bytes 2026-02-20T22:39:16.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1594241024 bytes 2026-02-20T22:39:16.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1673105408 bytes 2026-02-20T22:39:16.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1695291392 bytes 2026-02-20T22:39:16.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1741124608 bytes 2026-02-20T22:39:16.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1771113472 bytes 2026-02-20T22:39:16.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1786917888 bytes 2026-02-20T22:39:16.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1791468544 bytes 2026-02-20T22:39:16.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1863431168 bytes 2026-02-20T22:39:16.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1897864192 bytes 2026-02-20T22:39:16.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1922790400 bytes 2026-02-20T22:39:16.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1958714368 bytes 2026-02-20T22:39:16.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1985197056 bytes 2026-02-20T22:39:16.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1985832960 bytes 2026-02-20T22:39:16.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2018011136 bytes 2026-02-20T22:39:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2038197248 bytes 2026-02-20T22:39:16.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2073001984 bytes 2026-02-20T22:39:17.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2131858432 bytes 2026-02-20T22:39:17.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2166123520 bytes 2026-02-20T22:39:17.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2200130560 bytes 2026-02-20T22:39:17.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2210177024 bytes 2026-02-20T22:39:17.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2248326144 bytes 2026-02-20T22:39:17.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2308895744 bytes 2026-02-20T22:39:17.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2389199872 bytes 2026-02-20T22:39:17.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2402893824 bytes 2026-02-20T22:39:17.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2472367104 bytes 2026-02-20T22:39:17.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2513831936 bytes 2026-02-20T22:39:17.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2514446336 bytes 2026-02-20T22:39:17.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2614269952 bytes 2026-02-20T22:39:17.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2660886528 bytes 2026-02-20T22:39:17.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2688358400 bytes 2026-02-20T22:39:17.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2760502272 bytes 2026-02-20T22:39:17.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2770576384 bytes 2026-02-20T22:39:17.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2841614336 bytes 2026-02-20T22:39:18.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2875012096 bytes 2026-02-20T22:39:18.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2898598912 bytes 2026-02-20T22:39:18.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2930860032 bytes 2026-02-20T22:39:18.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2936367104 bytes 2026-02-20T22:39:18.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2998115328 bytes 2026-02-20T22:39:18.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3068520448 bytes 2026-02-20T22:39:18.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3148320768 bytes 2026-02-20T22:39:18.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3199423488 bytes 2026-02-20T22:39:18.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3257825280 bytes 2026-02-20T22:39:18.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3301096448 bytes 2026-02-20T22:39:18.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3398702080 bytes 2026-02-20T22:39:18.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3474714624 bytes 2026-02-20T22:39:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3531067392 bytes 2026-02-20T22:39:18.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3598396416 bytes 2026-02-20T22:39:18.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3665025024 bytes 2026-02-20T22:39:19.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3741204480 bytes 2026-02-20T22:39:19.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3795973120 bytes 2026-02-20T22:39:19.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3799358464 bytes 2026-02-20T22:39:19.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3833682944 bytes 2026-02-20T22:39:19.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3879268352 bytes 2026-02-20T22:39:19.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3879873536 bytes 2026-02-20T22:39:19.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3908926464 bytes 2026-02-20T22:39:19.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3936657408 bytes 2026-02-20T22:39:19.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3945880576 bytes 2026-02-20T22:39:19.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4044958720 bytes 2026-02-20T22:39:19.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4131927040 bytes 2026-02-20T22:39:19.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4180673536 bytes 2026-02-20T22:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:39:19.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_21 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_21 128M 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:39:19.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:19.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:19.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:19.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2026-02-20T22:39:20.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:39:20.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2026-02-20T22:39:20.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_21': 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13506a80b163 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13506a80b163 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:19 2026 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:19 2026 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:19 2026 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 69840726-9864-4961-8865-3713146736b0 2026-02-20T22:39:20.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_21 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:39:20.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:39:20.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:39:20.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:39:20.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:39:20.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 61176832 bytes 2026-02-20T22:39:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 71750656 bytes 2026-02-20T22:39:21.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 136430592 bytes 2026-02-20T22:39:21.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 146141184 bytes 2026-02-20T22:39:21.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 199895040 bytes 2026-02-20T22:39:21.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 247853056 bytes 2026-02-20T22:39:21.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 266098688 bytes 2026-02-20T22:39:21.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 330558464 bytes 2026-02-20T22:39:21.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 383176704 bytes 2026-02-20T22:39:21.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 412013568 bytes 2026-02-20T22:39:21.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 436298752 bytes 2026-02-20T22:39:21.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 555841536 bytes 2026-02-20T22:39:21.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 581819392 bytes 2026-02-20T22:39:21.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 615928832 bytes 2026-02-20T22:39:21.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 625830912 bytes 2026-02-20T22:39:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 664685568 bytes 2026-02-20T22:39:21.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 710446080 bytes 2026-02-20T22:39:21.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 719887360 bytes 2026-02-20T22:39:21.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 758343680 bytes 2026-02-20T22:39:21.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 861339648 bytes 2026-02-20T22:39:22.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 937660416 bytes 2026-02-20T22:39:22.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 967822336 bytes 2026-02-20T22:39:22.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1002192896 bytes 2026-02-20T22:39:22.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1091223552 bytes 2026-02-20T22:39:22.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1115545600 bytes 2026-02-20T22:39:22.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1172024320 bytes 2026-02-20T22:39:22.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1216721920 bytes 2026-02-20T22:39:22.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1258687488 bytes 2026-02-20T22:39:22.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1310123008 bytes 2026-02-20T22:39:22.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1339379712 bytes 2026-02-20T22:39:22.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1366405120 bytes 2026-02-20T22:39:22.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1376125952 bytes 2026-02-20T22:39:22.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1435196416 bytes 2026-02-20T22:39:22.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1461527552 bytes 2026-02-20T22:39:22.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1523257344 bytes 2026-02-20T22:39:22.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1569126400 bytes 2026-02-20T22:39:22.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1610179584 bytes 2026-02-20T22:39:23.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1653048320 bytes 2026-02-20T22:39:23.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1703636992 bytes 2026-02-20T22:39:23.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1788881920 bytes 2026-02-20T22:39:23.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1821263872 bytes 2026-02-20T22:39:23.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1854525440 bytes 2026-02-20T22:39:23.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1883585536 bytes 2026-02-20T22:39:23.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1958124544 bytes 2026-02-20T22:39:23.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1984856064 bytes 2026-02-20T22:39:23.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2001726464 bytes 2026-02-20T22:39:23.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2070910976 bytes 2026-02-20T22:39:23.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2127539200 bytes 2026-02-20T22:39:23.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2150770688 bytes 2026-02-20T22:39:23.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2181729280 bytes 2026-02-20T22:39:23.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2209825792 bytes 2026-02-20T22:39:23.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2278738944 bytes 2026-02-20T22:39:23.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2300784640 bytes 2026-02-20T22:39:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2350622720 bytes 2026-02-20T22:39:23.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2363189248 bytes 2026-02-20T22:39:23.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2363784192 bytes 2026-02-20T22:39:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2384702464 bytes 2026-02-20T22:39:24.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2430636032 bytes 2026-02-20T22:39:24.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2459032576 bytes 2026-02-20T22:39:24.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2496689152 bytes 2026-02-20T22:39:24.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2519951360 bytes 2026-02-20T22:39:24.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2583488512 bytes 2026-02-20T22:39:24.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2673428480 bytes 2026-02-20T22:39:24.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2714037248 bytes 2026-02-20T22:39:24.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2767757312 bytes 2026-02-20T22:39:24.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2791143424 bytes 2026-02-20T22:39:24.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2835249152 bytes 2026-02-20T22:39:24.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2884476928 bytes 2026-02-20T22:39:24.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2929118208 bytes 2026-02-20T22:39:24.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2949684224 bytes 2026-02-20T22:39:24.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2966791168 bytes 2026-02-20T22:39:24.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3011554304 bytes 2026-02-20T22:39:24.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3024262144 bytes 2026-02-20T22:39:24.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3070555136 bytes 2026-02-20T22:39:25.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3096904704 bytes 2026-02-20T22:39:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3149009920 bytes 2026-02-20T22:39:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3210469376 bytes 2026-02-20T22:39:25.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3231661056 bytes 2026-02-20T22:39:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3273225216 bytes 2026-02-20T22:39:25.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3331133440 bytes 2026-02-20T22:39:25.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3429945344 bytes 2026-02-20T22:39:25.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3460944896 bytes 2026-02-20T22:39:25.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3474618368 bytes 2026-02-20T22:39:25.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3504102400 bytes 2026-02-20T22:39:25.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3506689024 bytes 2026-02-20T22:39:25.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3545204736 bytes 2026-02-20T22:39:25.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3567260672 bytes 2026-02-20T22:39:25.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3581074432 bytes 2026-02-20T22:39:25.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3598972928 bytes 2026-02-20T22:39:25.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3636750336 bytes 2026-02-20T22:39:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3700923392 bytes 2026-02-20T22:39:25.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3731120128 bytes 2026-02-20T22:39:25.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3757986816 bytes 2026-02-20T22:39:26.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3827176448 bytes 2026-02-20T22:39:26.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3874582528 bytes 2026-02-20T22:39:26.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3925174272 bytes 2026-02-20T22:39:26.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3946547200 bytes 2026-02-20T22:39:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3994062848 bytes 2026-02-20T22:39:26.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4035832832 bytes 2026-02-20T22:39:26.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4095825920 bytes 2026-02-20T22:39:26.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4131015680 bytes 2026-02-20T22:39:26.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4152057856 bytes 2026-02-20T22:39:26.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4203758592 bytes 2026-02-20T22:39:26.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4270653440 bytes 2026-02-20T22:39:26.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4360222720 bytes 2026-02-20T22:39:26.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4374865920 bytes 2026-02-20T22:39:26.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4379729920 bytes 2026-02-20T22:39:26.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4410002432 bytes 2026-02-20T22:39:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4456062976 bytes 2026-02-20T22:39:26.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4501553152 bytes 2026-02-20T22:39:27.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4528974848 bytes 2026-02-20T22:39:27.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4605248512 bytes 2026-02-20T22:39:27.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4651090944 bytes 2026-02-20T22:39:27.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4691070976 bytes 2026-02-20T22:39:27.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4731088896 bytes 2026-02-20T22:39:27.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4769796096 bytes 2026-02-20T22:39:27.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4831050752 bytes 2026-02-20T22:39:27.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4848474112 bytes 2026-02-20T22:39:27.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4849081344 bytes 2026-02-20T22:39:27.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4895630336 bytes 2026-02-20T22:39:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4957474816 bytes 2026-02-20T22:39:27.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4978379776 bytes 2026-02-20T22:39:27.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5041405952 bytes 2026-02-20T22:39:27.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5105693696 bytes 2026-02-20T22:39:27.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5141220352 bytes 2026-02-20T22:39:27.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5146293248 bytes 2026-02-20T22:39:27.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5164995584 bytes 2026-02-20T22:39:27.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5240061952 bytes 2026-02-20T22:39:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5289661440 bytes 2026-02-20T22:39:28.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5368309760 bytes 2026-02-20T22:39:28.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5418553344 bytes 2026-02-20T22:39:28.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5481183232 bytes 2026-02-20T22:39:28.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5509126144 bytes 2026-02-20T22:39:28.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5565447168 bytes 2026-02-20T22:39:28.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5576114176 bytes 2026-02-20T22:39:28.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5598748672 bytes 2026-02-20T22:39:28.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5660179456 bytes 2026-02-20T22:39:28.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5676102656 bytes 2026-02-20T22:39:28.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5693021184 bytes 2026-02-20T22:39:28.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5718426624 bytes 2026-02-20T22:39:28.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5740011520 bytes 2026-02-20T22:39:28.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5755773952 bytes 2026-02-20T22:39:28.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5777503232 bytes 2026-02-20T22:39:28.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5839421440 bytes 2026-02-20T22:39:28.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5918900224 bytes 2026-02-20T22:39:29.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5989019648 bytes 2026-02-20T22:39:29.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6069864448 bytes 2026-02-20T22:39:29.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6153741312 bytes 2026-02-20T22:39:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6188905472 bytes 2026-02-20T22:39:29.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6205372416 bytes 2026-02-20T22:39:29.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6272444416 bytes 2026-02-20T22:39:29.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6273068032 bytes 2026-02-20T22:39:29.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6304259072 bytes 2026-02-20T22:39:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6322629632 bytes 2026-02-20T22:39:29.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6438945792 bytes 2026-02-20T22:39:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6476233728 bytes 2026-02-20T22:39:29.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6541403136 bytes 2026-02-20T22:39:29.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6570383360 bytes 2026-02-20T22:39:29.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6590434304 bytes 2026-02-20T22:39:29.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6662091776 bytes 2026-02-20T22:39:29.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6714710016 bytes 2026-02-20T22:39:29.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6759131136 bytes 2026-02-20T22:39:30.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6760438784 bytes 2026-02-20T22:39:30.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6849680384 bytes 2026-02-20T22:39:30.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6889641984 bytes 2026-02-20T22:39:30.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6935116800 bytes 2026-02-20T22:39:30.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6957326336 bytes 2026-02-20T22:39:30.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7014848512 bytes 2026-02-20T22:39:30.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7061383168 bytes 2026-02-20T22:39:30.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7093648384 bytes 2026-02-20T22:39:30.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7147241472 bytes 2026-02-20T22:39:30.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7167979520 bytes 2026-02-20T22:39:30.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7282870272 bytes 2026-02-20T22:39:30.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7330113536 bytes 2026-02-20T22:39:30.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7339229184 bytes 2026-02-20T22:39:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7427166208 bytes 2026-02-20T22:39:30.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7456928768 bytes 2026-02-20T22:39:30.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7497893888 bytes 2026-02-20T22:39:30.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7548608512 bytes 2026-02-20T22:39:31.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7587749888 bytes 2026-02-20T22:39:31.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7646223360 bytes 2026-02-20T22:39:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7661901824 bytes 2026-02-20T22:39:31.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7700117504 bytes 2026-02-20T22:39:31.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7758461952 bytes 2026-02-20T22:39:31.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7775544320 bytes 2026-02-20T22:39:31.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7833724928 bytes 2026-02-20T22:39:31.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7898983424 bytes 2026-02-20T22:39:31.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8024088576 bytes 2026-02-20T22:39:31.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8052722688 bytes 2026-02-20T22:39:31.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8156110848 bytes 2026-02-20T22:39:31.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8165892096 bytes 2026-02-20T22:39:31.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8166513664 bytes 2026-02-20T22:39:31.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8186711040 bytes 2026-02-20T22:39:31.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8243884032 bytes 2026-02-20T22:39:31.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8307721216 bytes 2026-02-20T22:39:31.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8309883904 bytes 2026-02-20T22:39:32.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8412251136 bytes 2026-02-20T22:39:32.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8448851968 bytes 2026-02-20T22:39:32.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8532441088 bytes 2026-02-20T22:39:32.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8598470656 bytes 2026-02-20T22:39:32.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8695027712 bytes 2026-02-20T22:39:32.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8756891648 bytes 2026-02-20T22:39:32.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8763922432 bytes 2026-02-20T22:39:32.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8841688064 bytes 2026-02-20T22:39:32.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8889064448 bytes 2026-02-20T22:39:32.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8924578816 bytes 2026-02-20T22:39:32.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8990785536 bytes 2026-02-20T22:39:32.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9034614784 bytes 2026-02-20T22:39:32.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9078200320 bytes 2026-02-20T22:39:32.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9089051648 bytes 2026-02-20T22:39:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9163132928 bytes 2026-02-20T22:39:32.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9208741888 bytes 2026-02-20T22:39:33.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9223023616 bytes 2026-02-20T22:39:33.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9250085888 bytes 2026-02-20T22:39:33.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9278350336 bytes 2026-02-20T22:39:33.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9330139136 bytes 2026-02-20T22:39:33.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9414238208 bytes 2026-02-20T22:39:33.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9449982976 bytes 2026-02-20T22:39:33.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9539680256 bytes 2026-02-20T22:39:33.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9622081536 bytes 2026-02-20T22:39:33.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9637140480 bytes 2026-02-20T22:39:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9687585792 bytes 2026-02-20T22:39:33.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9793045504 bytes 2026-02-20T22:39:33.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9821792256 bytes 2026-02-20T22:39:33.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9858590720 bytes 2026-02-20T22:39:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:39:33.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_22 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_22 128M 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2026-02-20T22:39:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:39:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:39:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2026-02-20T22:39:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:39:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:33.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2026-02-20T22:39:34.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:39:34.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_22': 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 135c6460f92a 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.135c6460f92a 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:39:34.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:33 2026 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:33 2026 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:33 2026 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 49060f6a-cc8c-4444-ac78-938dac73f064 2026-02-20T22:39:34.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:39:34.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:39:34.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_22 2026-02-20T22:39:34.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:34.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:34.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:39:34.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:39:34.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:39:34.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:39:34.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:39:34.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 23263232 bytes 2026-02-20T22:39:34.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 66123776 bytes 2026-02-20T22:39:34.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 103071744 bytes 2026-02-20T22:39:35.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 186198016 bytes 2026-02-20T22:39:35.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 263175168 bytes 2026-02-20T22:39:35.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 320116736 bytes 2026-02-20T22:39:35.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 353572864 bytes 2026-02-20T22:39:35.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 445549568 bytes 2026-02-20T22:39:35.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 533532672 bytes 2026-02-20T22:39:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 594205696 bytes 2026-02-20T22:39:35.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 656393216 bytes 2026-02-20T22:39:35.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 664901632 bytes 2026-02-20T22:39:35.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 695751680 bytes 2026-02-20T22:39:35.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 735476736 bytes 2026-02-20T22:39:35.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 759189504 bytes 2026-02-20T22:39:35.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 792187904 bytes 2026-02-20T22:39:35.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 867419136 bytes 2026-02-20T22:39:35.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 952466432 bytes 2026-02-20T22:39:36.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 1039913984 bytes 2026-02-20T22:39:36.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1098501120 bytes 2026-02-20T22:39:36.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1150082048 bytes 2026-02-20T22:39:36.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1195735040 bytes 2026-02-20T22:39:36.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1265844224 bytes 2026-02-20T22:39:36.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1354246144 bytes 2026-02-20T22:39:36.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1395639296 bytes 2026-02-20T22:39:36.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1416313856 bytes 2026-02-20T22:39:36.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1460294656 bytes 2026-02-20T22:39:36.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1467439104 bytes 2026-02-20T22:39:36.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1522695168 bytes 2026-02-20T22:39:36.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1535221760 bytes 2026-02-20T22:39:36.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1561857024 bytes 2026-02-20T22:39:36.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1605764096 bytes 2026-02-20T22:39:36.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1608505344 bytes 2026-02-20T22:39:36.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1669860352 bytes 2026-02-20T22:39:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1700622336 bytes 2026-02-20T22:39:37.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1752082432 bytes 2026-02-20T22:39:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1772506112 bytes 2026-02-20T22:39:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1833623552 bytes 2026-02-20T22:39:37.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1870380032 bytes 2026-02-20T22:39:37.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1925454848 bytes 2026-02-20T22:39:37.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1926056960 bytes 2026-02-20T22:39:37.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1954868224 bytes 2026-02-20T22:39:37.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1971784704 bytes 2026-02-20T22:39:37.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2019799040 bytes 2026-02-20T22:39:37.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2036990976 bytes 2026-02-20T22:39:37.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2106279936 bytes 2026-02-20T22:39:37.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2110562304 bytes 2026-02-20T22:39:37.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2145708032 bytes 2026-02-20T22:39:37.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2200510464 bytes 2026-02-20T22:39:37.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2237356032 bytes 2026-02-20T22:39:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2259641344 bytes 2026-02-20T22:39:37.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2292731904 bytes 2026-02-20T22:39:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2336141312 bytes 2026-02-20T22:39:38.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2341844992 bytes 2026-02-20T22:39:38.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2425639936 bytes 2026-02-20T22:39:38.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2486304768 bytes 2026-02-20T22:39:38.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2568791040 bytes 2026-02-20T22:39:38.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2630926336 bytes 2026-02-20T22:39:38.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2651997184 bytes 2026-02-20T22:39:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2676205568 bytes 2026-02-20T22:39:38.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2694302720 bytes 2026-02-20T22:39:38.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2733891584 bytes 2026-02-20T22:39:38.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2800288768 bytes 2026-02-20T22:39:38.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2848580608 bytes 2026-02-20T22:39:38.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2868571136 bytes 2026-02-20T22:39:38.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2953030656 bytes 2026-02-20T22:39:38.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3047701504 bytes 2026-02-20T22:39:38.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3102460928 bytes 2026-02-20T22:39:38.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3127350272 bytes 2026-02-20T22:39:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3156391936 bytes 2026-02-20T22:39:39.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3246960640 bytes 2026-02-20T22:39:39.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3342827520 bytes 2026-02-20T22:39:39.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3382968320 bytes 2026-02-20T22:39:39.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3423913984 bytes 2026-02-20T22:39:39.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3498845184 bytes 2026-02-20T22:39:39.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3561723904 bytes 2026-02-20T22:39:39.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3654467584 bytes 2026-02-20T22:39:39.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3705999360 bytes 2026-02-20T22:39:39.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3755061248 bytes 2026-02-20T22:39:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3791121408 bytes 2026-02-20T22:39:39.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3808832512 bytes 2026-02-20T22:39:39.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3848689664 bytes 2026-02-20T22:39:39.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3866469376 bytes 2026-02-20T22:39:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3929755648 bytes 2026-02-20T22:39:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3969143808 bytes 2026-02-20T22:39:40.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4016585728 bytes 2026-02-20T22:39:40.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4017185792 bytes 2026-02-20T22:39:40.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4094140416 bytes 2026-02-20T22:39:40.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4156133376 bytes 2026-02-20T22:39:40.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4163522560 bytes 2026-02-20T22:39:40.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4182678528 bytes 2026-02-20T22:39:40.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4209271808 bytes 2026-02-20T22:39:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4266826752 bytes 2026-02-20T22:39:40.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4368237568 bytes 2026-02-20T22:39:40.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4430158848 bytes 2026-02-20T22:39:40.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4470458368 bytes 2026-02-20T22:39:40.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4528710656 bytes 2026-02-20T22:39:40.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4575493120 bytes 2026-02-20T22:39:40.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4593595392 bytes 2026-02-20T22:39:40.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4611236864 bytes 2026-02-20T22:39:40.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4651689984 bytes 2026-02-20T22:39:41.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4691814400 bytes 2026-02-20T22:39:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4707315712 bytes 2026-02-20T22:39:41.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4752137216 bytes 2026-02-20T22:39:41.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4777004032 bytes 2026-02-20T22:39:41.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4785958912 bytes 2026-02-20T22:39:41.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4806881280 bytes 2026-02-20T22:39:41.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4963462144 bytes 2026-02-20T22:39:41.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 5026902016 bytes 2026-02-20T22:39:41.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5081350144 bytes 2026-02-20T22:39:41.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5130016768 bytes 2026-02-20T22:39:41.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5153339392 bytes 2026-02-20T22:39:41.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5207764992 bytes 2026-02-20T22:39:41.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5270403072 bytes 2026-02-20T22:39:41.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5332288512 bytes 2026-02-20T22:39:41.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5358566400 bytes 2026-02-20T22:39:42.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5445914624 bytes 2026-02-20T22:39:42.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5500169216 bytes 2026-02-20T22:39:42.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5556953088 bytes 2026-02-20T22:39:42.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5606372352 bytes 2026-02-20T22:39:42.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5678686208 bytes 2026-02-20T22:39:42.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5687335936 bytes 2026-02-20T22:39:42.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5721383936 bytes 2026-02-20T22:39:42.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5766414336 bytes 2026-02-20T22:39:42.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5805487104 bytes 2026-02-20T22:39:42.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5861293056 bytes 2026-02-20T22:39:42.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5888189440 bytes 2026-02-20T22:39:42.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5929098240 bytes 2026-02-20T22:39:42.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5988987904 bytes 2026-02-20T22:39:42.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6007834624 bytes 2026-02-20T22:39:42.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6063723520 bytes 2026-02-20T22:39:42.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6097473536 bytes 2026-02-20T22:39:42.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6108301312 bytes 2026-02-20T22:39:43.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6156538880 bytes 2026-02-20T22:39:43.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6173129728 bytes 2026-02-20T22:39:43.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6194086912 bytes 2026-02-20T22:39:43.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6266650624 bytes 2026-02-20T22:39:43.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6332490752 bytes 2026-02-20T22:39:43.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6379546624 bytes 2026-02-20T22:39:43.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6394164224 bytes 2026-02-20T22:39:43.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6434745344 bytes 2026-02-20T22:39:43.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6456915968 bytes 2026-02-20T22:39:43.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6489308160 bytes 2026-02-20T22:39:43.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6518907904 bytes 2026-02-20T22:39:43.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6525785088 bytes 2026-02-20T22:39:43.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6556532736 bytes 2026-02-20T22:39:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6598770688 bytes 2026-02-20T22:39:43.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6619544576 bytes 2026-02-20T22:39:43.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6660840448 bytes 2026-02-20T22:39:43.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6736902144 bytes 2026-02-20T22:39:44.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6786121728 bytes 2026-02-20T22:39:44.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6816887808 bytes 2026-02-20T22:39:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6866480128 bytes 2026-02-20T22:39:44.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6878436352 bytes 2026-02-20T22:39:44.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6960256000 bytes 2026-02-20T22:39:44.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6993755136 bytes 2026-02-20T22:39:44.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7045153792 bytes 2026-02-20T22:39:44.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7071097856 bytes 2026-02-20T22:39:44.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7098953728 bytes 2026-02-20T22:39:44.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7158708224 bytes 2026-02-20T22:39:44.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7210993664 bytes 2026-02-20T22:39:44.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7242564608 bytes 2026-02-20T22:39:44.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7278707712 bytes 2026-02-20T22:39:44.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7289675776 bytes 2026-02-20T22:39:44.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7333361664 bytes 2026-02-20T22:39:44.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7347828736 bytes 2026-02-20T22:39:44.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7392429056 bytes 2026-02-20T22:39:44.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7417024512 bytes 2026-02-20T22:39:44.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7427809280 bytes 2026-02-20T22:39:45.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7493679104 bytes 2026-02-20T22:39:45.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7509188608 bytes 2026-02-20T22:39:45.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7565587456 bytes 2026-02-20T22:39:45.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7611401216 bytes 2026-02-20T22:39:45.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7648822272 bytes 2026-02-20T22:39:45.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7675435008 bytes 2026-02-20T22:39:45.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7723613184 bytes 2026-02-20T22:39:45.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7737913344 bytes 2026-02-20T22:39:45.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7781882880 bytes 2026-02-20T22:39:45.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7839312896 bytes 2026-02-20T22:39:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7874410496 bytes 2026-02-20T22:39:45.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7892267008 bytes 2026-02-20T22:39:45.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7922608128 bytes 2026-02-20T22:39:46.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7976988672 bytes 2026-02-20T22:39:46.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8006000640 bytes 2026-02-20T22:39:46.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8022700032 bytes 2026-02-20T22:39:46.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8068304896 bytes 2026-02-20T22:39:46.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8074032128 bytes 2026-02-20T22:39:46.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8122529792 bytes 2026-02-20T22:39:46.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8168032256 bytes 2026-02-20T22:39:46.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8189720576 bytes 2026-02-20T22:39:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8289221632 bytes 2026-02-20T22:39:46.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8312127488 bytes 2026-02-20T22:39:46.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8398278656 bytes 2026-02-20T22:39:46.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8457349120 bytes 2026-02-20T22:39:46.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8457946112 bytes 2026-02-20T22:39:46.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8483044352 bytes 2026-02-20T22:39:46.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8557060096 bytes 2026-02-20T22:39:46.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8584178688 bytes 2026-02-20T22:39:46.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8632577024 bytes 2026-02-20T22:39:47.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8676788224 bytes 2026-02-20T22:39:47.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8727577600 bytes 2026-02-20T22:39:47.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8791103488 bytes 2026-02-20T22:39:47.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8804891648 bytes 2026-02-20T22:39:47.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8842273792 bytes 2026-02-20T22:39:47.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8902906880 bytes 2026-02-20T22:39:47.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8973030400 bytes 2026-02-20T22:39:47.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8998973440 bytes 2026-02-20T22:39:47.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9041737728 bytes 2026-02-20T22:39:47.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9103661056 bytes 2026-02-20T22:39:47.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9166067712 bytes 2026-02-20T22:39:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9219169280 bytes 2026-02-20T22:39:47.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9305648128 bytes 2026-02-20T22:39:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_23 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_23 128M 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:47.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2026-02-20T22:39:48.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:39:48.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2026-02-20T22:39:48.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_23': 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1368814ce38 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1368814ce38 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:39:48.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:47 2026 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:47 2026 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:47 2026 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 357d90ab-5f43-4a44-b85f-74b7bbc1d809 2026-02-20T22:39:48.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_23 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:39:48.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:39:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:39:48.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:39:48.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:39:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 32922624 bytes 2026-02-20T22:39:48.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 68905984 bytes 2026-02-20T22:39:49.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 116612096 bytes 2026-02-20T22:39:49.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 158961664 bytes 2026-02-20T22:39:49.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 204511232 bytes 2026-02-20T22:39:49.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 297888768 bytes 2026-02-20T22:39:49.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 303064064 bytes 2026-02-20T22:39:49.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 369666048 bytes 2026-02-20T22:39:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 430099456 bytes 2026-02-20T22:39:49.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 479180800 bytes 2026-02-20T22:39:49.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 490705920 bytes 2026-02-20T22:39:49.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 534948864 bytes 2026-02-20T22:39:49.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 595196928 bytes 2026-02-20T22:39:49.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 651300864 bytes 2026-02-20T22:39:49.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 706488320 bytes 2026-02-20T22:39:49.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 748157952 bytes 2026-02-20T22:39:49.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8139670954010162) > weight (0.25) 2026-02-20T22:39:49.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9935022219656857) > weight (0.25) 2026-02-20T22:39:49.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9831084323547395) > weight (0.25) 2026-02-20T22:39:49.870 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-20T22:39:49.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-20T22:39:49.871 INFO:tasks.rbd_mirror_thrash:waiting for 0.060332922500365704 secs before reviving daemons 2026-02-20T22:39:49.871 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T22:39:49.864+0000 7f603915a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 22358) UID: 1000 2026-02-20T22:39:49.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 806501376 bytes 2026-02-20T22:39:49.932 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-20T22:39:49.932 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:39:49.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 867253248 bytes 2026-02-20T22:39:49.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 874566656 bytes 2026-02-20T22:39:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 948616192 bytes 2026-02-20T22:39:50.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1002476544 bytes 2026-02-20T22:39:50.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1031080960 bytes 2026-02-20T22:39:50.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1073509376 bytes 2026-02-20T22:39:50.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1115695104 bytes 2026-02-20T22:39:50.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1157593088 bytes 2026-02-20T22:39:50.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1172663296 bytes 2026-02-20T22:39:50.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1228127232 bytes 2026-02-20T22:39:50.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1278530560 bytes 2026-02-20T22:39:50.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1295992832 bytes 2026-02-20T22:39:50.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1333699584 bytes 2026-02-20T22:39:50.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1378710528 bytes 2026-02-20T22:39:50.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1464645632 bytes 2026-02-20T22:39:50.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1570688000 bytes 2026-02-20T22:39:50.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1571318784 bytes 2026-02-20T22:39:50.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1624943616 bytes 2026-02-20T22:39:50.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1648833536 bytes 2026-02-20T22:39:51.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1709247488 bytes 2026-02-20T22:39:51.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1738722304 bytes 2026-02-20T22:39:51.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1802152960 bytes 2026-02-20T22:39:51.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1840859136 bytes 2026-02-20T22:39:51.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1866173440 bytes 2026-02-20T22:39:51.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1945168896 bytes 2026-02-20T22:39:51.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1980738560 bytes 2026-02-20T22:39:51.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1981346816 bytes 2026-02-20T22:39:51.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2057299968 bytes 2026-02-20T22:39:51.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2119552000 bytes 2026-02-20T22:39:51.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2185877504 bytes 2026-02-20T22:39:51.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2246061056 bytes 2026-02-20T22:39:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2250213376 bytes 2026-02-20T22:39:51.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2294194176 bytes 2026-02-20T22:39:51.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2336571392 bytes 2026-02-20T22:39:51.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2378210304 bytes 2026-02-20T22:39:51.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2409069568 bytes 2026-02-20T22:39:52.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2468319232 bytes 2026-02-20T22:39:52.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2527447040 bytes 2026-02-20T22:39:52.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2613277696 bytes 2026-02-20T22:39:52.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2613900288 bytes 2026-02-20T22:39:52.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2680925184 bytes 2026-02-20T22:39:52.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2713422848 bytes 2026-02-20T22:39:52.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2744948736 bytes 2026-02-20T22:39:52.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2777732096 bytes 2026-02-20T22:39:52.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2825633792 bytes 2026-02-20T22:39:52.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2886901760 bytes 2026-02-20T22:39:52.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2911128576 bytes 2026-02-20T22:39:52.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2924195840 bytes 2026-02-20T22:39:52.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2979009536 bytes 2026-02-20T22:39:52.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3019346944 bytes 2026-02-20T22:39:52.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3049849856 bytes 2026-02-20T22:39:52.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3060710400 bytes 2026-02-20T22:39:52.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3077279744 bytes 2026-02-20T22:39:52.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3127014400 bytes 2026-02-20T22:39:53.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3210099712 bytes 2026-02-20T22:39:53.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3307042816 bytes 2026-02-20T22:39:53.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3329589248 bytes 2026-02-20T22:39:53.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3360617472 bytes 2026-02-20T22:39:53.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3375478784 bytes 2026-02-20T22:39:53.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3410106368 bytes 2026-02-20T22:39:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3431798784 bytes 2026-02-20T22:39:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3479644160 bytes 2026-02-20T22:39:53.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3510474752 bytes 2026-02-20T22:39:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3556189184 bytes 2026-02-20T22:39:53.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3570311168 bytes 2026-02-20T22:39:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3631490048 bytes 2026-02-20T22:39:53.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3670566912 bytes 2026-02-20T22:39:53.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3728159744 bytes 2026-02-20T22:39:53.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3788340224 bytes 2026-02-20T22:39:53.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3837922304 bytes 2026-02-20T22:39:53.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3845046272 bytes 2026-02-20T22:39:53.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3857379328 bytes 2026-02-20T22:39:54.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3987592192 bytes 2026-02-20T22:39:54.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4054279168 bytes 2026-02-20T22:39:54.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4087746560 bytes 2026-02-20T22:39:54.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4143289344 bytes 2026-02-20T22:39:54.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4200615936 bytes 2026-02-20T22:39:54.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4238900224 bytes 2026-02-20T22:39:54.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4254993408 bytes 2026-02-20T22:39:54.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4299362304 bytes 2026-02-20T22:39:54.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4407027712 bytes 2026-02-20T22:39:54.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4468062208 bytes 2026-02-20T22:39:54.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4533984256 bytes 2026-02-20T22:39:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4562349056 bytes 2026-02-20T22:39:54.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4563001344 bytes 2026-02-20T22:39:54.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4581652480 bytes 2026-02-20T22:39:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:39:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:39:54.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_24 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:54.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_24 128M 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:39:54.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:39:54.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2026-02-20T22:39:54.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:39:54.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:54.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:39:54.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:39:54.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2026-02-20T22:39:55.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:39:55.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2026-02-20T22:39:55.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_24': 2026-02-20T22:39:55.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:39:55.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:39:55.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:39:55.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13743346674 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13743346674 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:54 2026 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:54 2026 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:54 2026 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: eaa9a84f-e88b-4266-b522-69fac662b35d 2026-02-20T22:39:55.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:39:55.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:39:55.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_24 2026-02-20T22:39:55.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:39:55.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:39:55.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:39:55.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:39:55.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:39:55.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:39:55.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:39:55.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 19803136 bytes 2026-02-20T22:39:56.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 90838016 bytes 2026-02-20T22:39:56.034 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-20T22:39:56.034 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:39:56.034 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:39:56.036 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:39:56.036 INFO:tasks.rbd_mirror_thrash:waiting for 65.48305659399098 secs before thrashing 2026-02-20T22:39:56.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 148144128 bytes 2026-02-20T22:39:56.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 232810496 bytes 2026-02-20T22:39:56.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 248325120 bytes 2026-02-20T22:39:56.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 310070272 bytes 2026-02-20T22:39:56.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 334689280 bytes 2026-02-20T22:39:56.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 349557760 bytes 2026-02-20T22:39:56.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 370644992 bytes 2026-02-20T22:39:56.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 459624448 bytes 2026-02-20T22:39:56.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 536523776 bytes 2026-02-20T22:39:56.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 561155072 bytes 2026-02-20T22:39:56.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 587985920 bytes 2026-02-20T22:39:56.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 627099648 bytes 2026-02-20T22:39:56.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 657018880 bytes 2026-02-20T22:39:56.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 690947072 bytes 2026-02-20T22:39:56.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 717299712 bytes 2026-02-20T22:39:56.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 770212864 bytes 2026-02-20T22:39:56.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 828133376 bytes 2026-02-20T22:39:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 895667200 bytes 2026-02-20T22:39:57.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 939791360 bytes 2026-02-20T22:39:57.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 944017408 bytes 2026-02-20T22:39:57.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 961045504 bytes 2026-02-20T22:39:57.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 975675392 bytes 2026-02-20T22:39:57.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 986177536 bytes 2026-02-20T22:39:57.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1006492672 bytes 2026-02-20T22:39:57.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1045219328 bytes 2026-02-20T22:39:57.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1065906176 bytes 2026-02-20T22:39:57.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1097420800 bytes 2026-02-20T22:39:57.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1118768128 bytes 2026-02-20T22:39:57.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1229737984 bytes 2026-02-20T22:39:57.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1244430336 bytes 2026-02-20T22:39:57.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1330789376 bytes 2026-02-20T22:39:57.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1354645504 bytes 2026-02-20T22:39:57.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1406592000 bytes 2026-02-20T22:39:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1484048384 bytes 2026-02-20T22:39:57.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1527933952 bytes 2026-02-20T22:39:57.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1563657216 bytes 2026-02-20T22:39:58.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1595217920 bytes 2026-02-20T22:39:58.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1698618368 bytes 2026-02-20T22:39:58.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1734831104 bytes 2026-02-20T22:39:58.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1796900864 bytes 2026-02-20T22:39:58.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1799735296 bytes 2026-02-20T22:39:58.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1818403840 bytes 2026-02-20T22:39:58.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1829929984 bytes 2026-02-20T22:39:58.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1896953856 bytes 2026-02-20T22:39:58.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1934200832 bytes 2026-02-20T22:39:58.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2021146624 bytes 2026-02-20T22:39:58.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2037489664 bytes 2026-02-20T22:39:58.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2092998656 bytes 2026-02-20T22:39:58.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2124278784 bytes 2026-02-20T22:39:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2147210240 bytes 2026-02-20T22:39:58.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2147816448 bytes 2026-02-20T22:39:58.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2231640064 bytes 2026-02-20T22:39:58.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2301938688 bytes 2026-02-20T22:39:59.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2323359744 bytes 2026-02-20T22:39:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2434356224 bytes 2026-02-20T22:39:59.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2454820864 bytes 2026-02-20T22:39:59.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2509997056 bytes 2026-02-20T22:39:59.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2584466432 bytes 2026-02-20T22:39:59.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2630414336 bytes 2026-02-20T22:39:59.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2653101056 bytes 2026-02-20T22:39:59.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2696216576 bytes 2026-02-20T22:39:59.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2716815360 bytes 2026-02-20T22:39:59.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2776536064 bytes 2026-02-20T22:39:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2824241152 bytes 2026-02-20T22:39:59.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2894274560 bytes 2026-02-20T22:39:59.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2916816896 bytes 2026-02-20T22:39:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2928542720 bytes 2026-02-20T22:39:59.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2949482496 bytes 2026-02-20T22:39:59.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3006774272 bytes 2026-02-20T22:39:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3040782336 bytes 2026-02-20T22:40:00.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3098503168 bytes 2026-02-20T22:40:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3187671040 bytes 2026-02-20T22:40:00.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3283122176 bytes 2026-02-20T22:40:00.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3299536896 bytes 2026-02-20T22:40:00.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3319805952 bytes 2026-02-20T22:40:00.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3320406016 bytes 2026-02-20T22:40:00.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3390105600 bytes 2026-02-20T22:40:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3421024256 bytes 2026-02-20T22:40:00.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3463644160 bytes 2026-02-20T22:40:00.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3488309248 bytes 2026-02-20T22:40:00.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3505951744 bytes 2026-02-20T22:40:00.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3529244672 bytes 2026-02-20T22:40:00.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3592454144 bytes 2026-02-20T22:40:00.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3649705984 bytes 2026-02-20T22:40:00.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3683348480 bytes 2026-02-20T22:40:00.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3728405504 bytes 2026-02-20T22:40:00.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3763799040 bytes 2026-02-20T22:40:00.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3806903296 bytes 2026-02-20T22:40:01.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3853355008 bytes 2026-02-20T22:40:01.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3883812864 bytes 2026-02-20T22:40:01.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3914420224 bytes 2026-02-20T22:40:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3931269120 bytes 2026-02-20T22:40:01.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3988367360 bytes 2026-02-20T22:40:01.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4011383808 bytes 2026-02-20T22:40:01.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4042592256 bytes 2026-02-20T22:40:01.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4101258240 bytes 2026-02-20T22:40:01.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4144935936 bytes 2026-02-20T22:40:01.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4197985280 bytes 2026-02-20T22:40:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4267574272 bytes 2026-02-20T22:40:01.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4276795392 bytes 2026-02-20T22:40:01.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4349045760 bytes 2026-02-20T22:40:01.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4379150336 bytes 2026-02-20T22:40:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4390467584 bytes 2026-02-20T22:40:01.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4422022144 bytes 2026-02-20T22:40:01.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4475299840 bytes 2026-02-20T22:40:01.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4496409600 bytes 2026-02-20T22:40:01.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4577089536 bytes 2026-02-20T22:40:02.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4647994368 bytes 2026-02-20T22:40:02.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4677030912 bytes 2026-02-20T22:40:02.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4677639168 bytes 2026-02-20T22:40:02.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4690085888 bytes 2026-02-20T22:40:02.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4746277888 bytes 2026-02-20T22:40:02.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4759930880 bytes 2026-02-20T22:40:02.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4785968128 bytes 2026-02-20T22:40:02.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4817005568 bytes 2026-02-20T22:40:02.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4861078528 bytes 2026-02-20T22:40:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4904028160 bytes 2026-02-20T22:40:02.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4941344768 bytes 2026-02-20T22:40:02.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4953777152 bytes 2026-02-20T22:40:02.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4970204160 bytes 2026-02-20T22:40:02.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5003790336 bytes 2026-02-20T22:40:02.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5021331456 bytes 2026-02-20T22:40:02.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5118368768 bytes 2026-02-20T22:40:02.870 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9812774820999473) > weight (0.25) 2026-02-20T22:40:02.871 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.43900488215809963) > weight (0.25) 2026-02-20T22:40:02.871 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.25413445554740755) > weight (0.25) 2026-02-20T22:40:02.871 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2026-02-20T22:40:02.871 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2026-02-20T22:40:02.871 INFO:tasks.rbd_mirror_thrash:waiting for 7.109221405139797 secs before reviving daemons 2026-02-20T22:40:02.872 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:40:02.864+0000 7f2ab81b3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 24751) UID: 1000 2026-02-20T22:40:02.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5165217792 bytes 2026-02-20T22:40:02.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5248941056 bytes 2026-02-20T22:40:03.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5274422272 bytes 2026-02-20T22:40:03.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5305329664 bytes 2026-02-20T22:40:03.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5359969280 bytes 2026-02-20T22:40:03.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5432000512 bytes 2026-02-20T22:40:03.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5458428928 bytes 2026-02-20T22:40:03.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5498699776 bytes 2026-02-20T22:40:03.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5499320320 bytes 2026-02-20T22:40:03.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5556457472 bytes 2026-02-20T22:40:03.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5605017600 bytes 2026-02-20T22:40:03.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5622557696 bytes 2026-02-20T22:40:03.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5739804672 bytes 2026-02-20T22:40:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5773151232 bytes 2026-02-20T22:40:03.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5805647872 bytes 2026-02-20T22:40:03.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5817858048 bytes 2026-02-20T22:40:03.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5842505728 bytes 2026-02-20T22:40:03.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5894547456 bytes 2026-02-20T22:40:03.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5939400704 bytes 2026-02-20T22:40:03.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5965426688 bytes 2026-02-20T22:40:04.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6073713664 bytes 2026-02-20T22:40:04.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6182296576 bytes 2026-02-20T22:40:04.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6202946560 bytes 2026-02-20T22:40:04.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6247885824 bytes 2026-02-20T22:40:04.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6303754240 bytes 2026-02-20T22:40:04.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6318901248 bytes 2026-02-20T22:40:04.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6358258688 bytes 2026-02-20T22:40:04.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6389131264 bytes 2026-02-20T22:40:04.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6428040192 bytes 2026-02-20T22:40:04.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6489510912 bytes 2026-02-20T22:40:04.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6525536256 bytes 2026-02-20T22:40:04.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6552166400 bytes 2026-02-20T22:40:04.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6570830848 bytes 2026-02-20T22:40:04.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6624750592 bytes 2026-02-20T22:40:04.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6683566080 bytes 2026-02-20T22:40:04.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6738228224 bytes 2026-02-20T22:40:05.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6794642432 bytes 2026-02-20T22:40:05.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6806587392 bytes 2026-02-20T22:40:05.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6829198336 bytes 2026-02-20T22:40:05.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6876454912 bytes 2026-02-20T22:40:05.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6898191360 bytes 2026-02-20T22:40:05.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6957300736 bytes 2026-02-20T22:40:05.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7003216896 bytes 2026-02-20T22:40:05.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7042045952 bytes 2026-02-20T22:40:05.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7065025536 bytes 2026-02-20T22:40:05.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7158082560 bytes 2026-02-20T22:40:05.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7181166592 bytes 2026-02-20T22:40:05.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7238518784 bytes 2026-02-20T22:40:05.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7303096320 bytes 2026-02-20T22:40:05.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7343756288 bytes 2026-02-20T22:40:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7374157824 bytes 2026-02-20T22:40:05.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7387772928 bytes 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_25 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:05.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_25 128M 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:05.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2026-02-20T22:40:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:05.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:05.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2026-02-20T22:40:06.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:06.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_25': 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1386c63133a 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1386c63133a 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:06.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:05 2026 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:05 2026 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:05 2026 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 07156c03-24cc-4962-b7eb-f044afb9ccd1 2026-02-20T22:40:06.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_25 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:40:06.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:06.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=8 2026-02-20T22:40:06.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:06.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:06.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 84897792 bytes 2026-02-20T22:40:07.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 119358464 bytes 2026-02-20T22:40:07.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 153162752 bytes 2026-02-20T22:40:07.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 189275136 bytes 2026-02-20T22:40:07.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 234804224 bytes 2026-02-20T22:40:07.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 292385792 bytes 2026-02-20T22:40:07.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 374946816 bytes 2026-02-20T22:40:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 392121344 bytes 2026-02-20T22:40:07.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 432636928 bytes 2026-02-20T22:40:07.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 465878016 bytes 2026-02-20T22:40:07.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 501127168 bytes 2026-02-20T22:40:07.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 520397824 bytes 2026-02-20T22:40:07.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 580506624 bytes 2026-02-20T22:40:07.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 619339776 bytes 2026-02-20T22:40:07.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 667658240 bytes 2026-02-20T22:40:07.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 748525568 bytes 2026-02-20T22:40:07.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 882058240 bytes 2026-02-20T22:40:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 925492224 bytes 2026-02-20T22:40:08.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 975610880 bytes 2026-02-20T22:40:08.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1056033792 bytes 2026-02-20T22:40:08.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1080451072 bytes 2026-02-20T22:40:08.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1081085952 bytes 2026-02-20T22:40:08.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1138747392 bytes 2026-02-20T22:40:08.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1175967744 bytes 2026-02-20T22:40:08.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1176822784 bytes 2026-02-20T22:40:08.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1213742080 bytes 2026-02-20T22:40:08.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1295967232 bytes 2026-02-20T22:40:08.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1299947520 bytes 2026-02-20T22:40:08.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1327739904 bytes 2026-02-20T22:40:08.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1339134976 bytes 2026-02-20T22:40:08.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1360227328 bytes 2026-02-20T22:40:08.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1401119744 bytes 2026-02-20T22:40:08.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1442045952 bytes 2026-02-20T22:40:08.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1454182400 bytes 2026-02-20T22:40:08.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1456033792 bytes 2026-02-20T22:40:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1497772032 bytes 2026-02-20T22:40:08.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1517740032 bytes 2026-02-20T22:40:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1594425344 bytes 2026-02-20T22:40:08.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1597848576 bytes 2026-02-20T22:40:09.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1644789760 bytes 2026-02-20T22:40:09.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1669045248 bytes 2026-02-20T22:40:09.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1680218112 bytes 2026-02-20T22:40:09.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1707445248 bytes 2026-02-20T22:40:09.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1728882688 bytes 2026-02-20T22:40:09.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1757724672 bytes 2026-02-20T22:40:09.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1799257088 bytes 2026-02-20T22:40:09.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1834817536 bytes 2026-02-20T22:40:09.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1941339136 bytes 2026-02-20T22:40:09.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2012889088 bytes 2026-02-20T22:40:09.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2038286336 bytes 2026-02-20T22:40:09.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2093334528 bytes 2026-02-20T22:40:09.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2154607616 bytes 2026-02-20T22:40:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2189920256 bytes 2026-02-20T22:40:09.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2245051392 bytes 2026-02-20T22:40:09.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2245694464 bytes 2026-02-20T22:40:09.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2304851968 bytes 2026-02-20T22:40:09.981 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2026-02-20T22:40:09.981 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:40:09.982 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2026-02-20T22:40:09.982 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2026-02-20T22:40:09.982 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2026-02-20T22:40:09.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2026-02-20T22:40:09.983 INFO:tasks.rbd_mirror_thrash:waiting for 73.55485325878519 secs before thrashing 2026-02-20T22:40:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2328829952 bytes 2026-02-20T22:40:10.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2333101056 bytes 2026-02-20T22:40:10.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2413788160 bytes 2026-02-20T22:40:10.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2432927744 bytes 2026-02-20T22:40:10.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2462126080 bytes 2026-02-20T22:40:10.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2482246656 bytes 2026-02-20T22:40:10.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2530691072 bytes 2026-02-20T22:40:10.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2559935488 bytes 2026-02-20T22:40:10.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2647333888 bytes 2026-02-20T22:40:10.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2647951360 bytes 2026-02-20T22:40:10.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2701442048 bytes 2026-02-20T22:40:10.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2703045632 bytes 2026-02-20T22:40:10.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2712348672 bytes 2026-02-20T22:40:10.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2727189504 bytes 2026-02-20T22:40:10.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2775223296 bytes 2026-02-20T22:40:10.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2841524224 bytes 2026-02-20T22:40:10.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2866372608 bytes 2026-02-20T22:40:10.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2874724352 bytes 2026-02-20T22:40:10.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2882139136 bytes 2026-02-20T22:40:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2937457664 bytes 2026-02-20T22:40:10.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2976111616 bytes 2026-02-20T22:40:11.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3033044992 bytes 2026-02-20T22:40:11.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3059102720 bytes 2026-02-20T22:40:11.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3087943680 bytes 2026-02-20T22:40:11.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3153325056 bytes 2026-02-20T22:40:11.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3190212608 bytes 2026-02-20T22:40:11.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3228598272 bytes 2026-02-20T22:40:11.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3244754944 bytes 2026-02-20T22:40:11.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3250836480 bytes 2026-02-20T22:40:11.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3292833792 bytes 2026-02-20T22:40:11.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3336068096 bytes 2026-02-20T22:40:11.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3388773376 bytes 2026-02-20T22:40:11.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3426833408 bytes 2026-02-20T22:40:11.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3455836160 bytes 2026-02-20T22:40:11.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3546213376 bytes 2026-02-20T22:40:11.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3563701248 bytes 2026-02-20T22:40:11.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3583375360 bytes 2026-02-20T22:40:11.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3684300800 bytes 2026-02-20T22:40:11.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3745277952 bytes 2026-02-20T22:40:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3781584896 bytes 2026-02-20T22:40:12.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3791983616 bytes 2026-02-20T22:40:12.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3859493888 bytes 2026-02-20T22:40:12.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3892334592 bytes 2026-02-20T22:40:12.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3926546432 bytes 2026-02-20T22:40:12.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3932447744 bytes 2026-02-20T22:40:12.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4008404992 bytes 2026-02-20T22:40:12.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4029375488 bytes 2026-02-20T22:40:12.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4061790208 bytes 2026-02-20T22:40:12.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4097580032 bytes 2026-02-20T22:40:12.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4201226240 bytes 2026-02-20T22:40:12.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4225781760 bytes 2026-02-20T22:40:12.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4253238272 bytes 2026-02-20T22:40:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4296994816 bytes 2026-02-20T22:40:12.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4351556608 bytes 2026-02-20T22:40:12.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4394831872 bytes 2026-02-20T22:40:12.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4396975104 bytes 2026-02-20T22:40:12.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4438973440 bytes 2026-02-20T22:40:13.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4471335936 bytes 2026-02-20T22:40:13.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4526306304 bytes 2026-02-20T22:40:13.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4600861696 bytes 2026-02-20T22:40:13.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4629550080 bytes 2026-02-20T22:40:13.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4665709568 bytes 2026-02-20T22:40:13.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4719174656 bytes 2026-02-20T22:40:13.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4739529728 bytes 2026-02-20T22:40:13.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4778416128 bytes 2026-02-20T22:40:13.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4826092544 bytes 2026-02-20T22:40:13.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4860524544 bytes 2026-02-20T22:40:13.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4881316864 bytes 2026-02-20T22:40:13.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4897229824 bytes 2026-02-20T22:40:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4897833984 bytes 2026-02-20T22:40:13.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 4975115264 bytes 2026-02-20T22:40:13.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5042418688 bytes 2026-02-20T22:40:13.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5106134016 bytes 2026-02-20T22:40:13.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5138000896 bytes 2026-02-20T22:40:13.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5211584512 bytes 2026-02-20T22:40:14.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5231059968 bytes 2026-02-20T22:40:14.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5321421824 bytes 2026-02-20T22:40:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5386284032 bytes 2026-02-20T22:40:14.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5386896384 bytes 2026-02-20T22:40:14.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5420434432 bytes 2026-02-20T22:40:14.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5439453184 bytes 2026-02-20T22:40:14.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5472975872 bytes 2026-02-20T22:40:14.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5530378240 bytes 2026-02-20T22:40:14.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5615266816 bytes 2026-02-20T22:40:14.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5671520256 bytes 2026-02-20T22:40:14.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5707656192 bytes 2026-02-20T22:40:14.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5795865600 bytes 2026-02-20T22:40:14.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5831564288 bytes 2026-02-20T22:40:14.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5910966272 bytes 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_26 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_26 128M 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2026-02-20T22:40:14.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:14.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:14.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:14.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:14.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2026-02-20T22:40:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:15.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_26': 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 139bf8c84af1 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.139bf8c84af1 2026-02-20T22:40:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:14 2026 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:14 2026 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:14 2026 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 96d0ea21-daaa-43c4-b57c-6d4a1854bd5a 2026-02-20T22:40:15.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:15.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:15.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_26 2026-02-20T22:40:15.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:15.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:15.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:40:15.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:15.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:40:15.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:15.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:15.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 38755328 bytes 2026-02-20T22:40:16.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 52213760 bytes 2026-02-20T22:40:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 96235520 bytes 2026-02-20T22:40:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 192720896 bytes 2026-02-20T22:40:16.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 256766976 bytes 2026-02-20T22:40:16.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 290838528 bytes 2026-02-20T22:40:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 315883520 bytes 2026-02-20T22:40:16.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 377614336 bytes 2026-02-20T22:40:16.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 434661376 bytes 2026-02-20T22:40:16.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 459493376 bytes 2026-02-20T22:40:16.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 496638976 bytes 2026-02-20T22:40:16.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 520983552 bytes 2026-02-20T22:40:16.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 557899776 bytes 2026-02-20T22:40:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 569640960 bytes 2026-02-20T22:40:16.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 642217984 bytes 2026-02-20T22:40:16.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 670695424 bytes 2026-02-20T22:40:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 767441920 bytes 2026-02-20T22:40:16.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 855054336 bytes 2026-02-20T22:40:17.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 860231680 bytes 2026-02-20T22:40:17.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 866329600 bytes 2026-02-20T22:40:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 928219136 bytes 2026-02-20T22:40:17.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 942811136 bytes 2026-02-20T22:40:17.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1044352000 bytes 2026-02-20T22:40:17.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1049452544 bytes 2026-02-20T22:40:17.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1085191168 bytes 2026-02-20T22:40:17.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1108475904 bytes 2026-02-20T22:40:17.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1119862784 bytes 2026-02-20T22:40:17.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1149129728 bytes 2026-02-20T22:40:17.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1163166720 bytes 2026-02-20T22:40:17.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1232372736 bytes 2026-02-20T22:40:17.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1289367552 bytes 2026-02-20T22:40:17.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1310139392 bytes 2026-02-20T22:40:17.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1344151552 bytes 2026-02-20T22:40:17.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1397346304 bytes 2026-02-20T22:40:17.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1443771392 bytes 2026-02-20T22:40:17.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1474724864 bytes 2026-02-20T22:40:17.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1484542976 bytes 2026-02-20T22:40:18.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1542679552 bytes 2026-02-20T22:40:18.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1545758720 bytes 2026-02-20T22:40:18.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1619879936 bytes 2026-02-20T22:40:18.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1645656064 bytes 2026-02-20T22:40:18.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1781491712 bytes 2026-02-20T22:40:18.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1833904128 bytes 2026-02-20T22:40:18.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1898391552 bytes 2026-02-20T22:40:18.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1919789056 bytes 2026-02-20T22:40:18.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1957304320 bytes 2026-02-20T22:40:18.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2001899520 bytes 2026-02-20T22:40:18.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2022596608 bytes 2026-02-20T22:40:18.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2066490368 bytes 2026-02-20T22:40:18.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2124233728 bytes 2026-02-20T22:40:18.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2168202240 bytes 2026-02-20T22:40:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2213602304 bytes 2026-02-20T22:40:18.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2256945152 bytes 2026-02-20T22:40:19.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2291018752 bytes 2026-02-20T22:40:19.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2334129152 bytes 2026-02-20T22:40:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2398229504 bytes 2026-02-20T22:40:19.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2470319104 bytes 2026-02-20T22:40:19.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2547273728 bytes 2026-02-20T22:40:19.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2633591808 bytes 2026-02-20T22:40:19.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2674157568 bytes 2026-02-20T22:40:19.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2702497792 bytes 2026-02-20T22:40:19.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2747427840 bytes 2026-02-20T22:40:19.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2800880640 bytes 2026-02-20T22:40:19.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2867032064 bytes 2026-02-20T22:40:19.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2962590720 bytes 2026-02-20T22:40:19.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3011422208 bytes 2026-02-20T22:40:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3053753344 bytes 2026-02-20T22:40:19.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3110633472 bytes 2026-02-20T22:40:20.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3180358656 bytes 2026-02-20T22:40:20.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3246577664 bytes 2026-02-20T22:40:20.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3290265600 bytes 2026-02-20T22:40:20.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3290901504 bytes 2026-02-20T22:40:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3316085760 bytes 2026-02-20T22:40:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3339616256 bytes 2026-02-20T22:40:20.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3340247040 bytes 2026-02-20T22:40:20.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3380520960 bytes 2026-02-20T22:40:20.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3438144512 bytes 2026-02-20T22:40:20.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3447622656 bytes 2026-02-20T22:40:20.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3571151872 bytes 2026-02-20T22:40:20.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3619864576 bytes 2026-02-20T22:40:20.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3642968064 bytes 2026-02-20T22:40:20.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3657170944 bytes 2026-02-20T22:40:20.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3757903872 bytes 2026-02-20T22:40:20.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3765087232 bytes 2026-02-20T22:40:20.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3778123776 bytes 2026-02-20T22:40:20.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3828474880 bytes 2026-02-20T22:40:21.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3873071104 bytes 2026-02-20T22:40:21.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3912758272 bytes 2026-02-20T22:40:21.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3920868352 bytes 2026-02-20T22:40:21.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4002803712 bytes 2026-02-20T22:40:21.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4092610560 bytes 2026-02-20T22:40:21.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4169878528 bytes 2026-02-20T22:40:21.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4177039360 bytes 2026-02-20T22:40:21.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4204656640 bytes 2026-02-20T22:40:21.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4232535040 bytes 2026-02-20T22:40:21.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4273220608 bytes 2026-02-20T22:40:21.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4342330368 bytes 2026-02-20T22:40:21.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4389186560 bytes 2026-02-20T22:40:21.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4437501952 bytes 2026-02-20T22:40:21.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4551772160 bytes 2026-02-20T22:40:21.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4603353088 bytes 2026-02-20T22:40:21.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4609826816 bytes 2026-02-20T22:40:21.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4643379200 bytes 2026-02-20T22:40:22.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4699693056 bytes 2026-02-20T22:40:22.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4800068608 bytes 2026-02-20T22:40:22.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4869382144 bytes 2026-02-20T22:40:22.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4889693184 bytes 2026-02-20T22:40:22.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4923143168 bytes 2026-02-20T22:40:22.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4967632896 bytes 2026-02-20T22:40:22.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4984759296 bytes 2026-02-20T22:40:22.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5024443392 bytes 2026-02-20T22:40:22.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5084421120 bytes 2026-02-20T22:40:22.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5175115776 bytes 2026-02-20T22:40:22.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5224220672 bytes 2026-02-20T22:40:22.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5314949120 bytes 2026-02-20T22:40:22.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5391733760 bytes 2026-02-20T22:40:22.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5398474752 bytes 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:22.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_27 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_27 128M 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:22.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:22.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2026-02-20T22:40:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:23.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2026-02-20T22:40:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_27': 2026-02-20T22:40:23.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13a4ea0aeb93 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13a4ea0aeb93 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:22 2026 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:22 2026 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:22 2026 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9dacbf76-f085-4405-b87d-fe7a44b68efb 2026-02-20T22:40:23.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_27 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:40:23.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:23.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:40:23.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:23.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:23.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 64480256 bytes 2026-02-20T22:40:24.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 86005760 bytes 2026-02-20T22:40:24.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 147162112 bytes 2026-02-20T22:40:24.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 197068800 bytes 2026-02-20T22:40:24.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 201934848 bytes 2026-02-20T22:40:24.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 249629696 bytes 2026-02-20T22:40:24.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 250227712 bytes 2026-02-20T22:40:24.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 292835328 bytes 2026-02-20T22:40:24.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 357051392 bytes 2026-02-20T22:40:24.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 384779264 bytes 2026-02-20T22:40:24.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 433873920 bytes 2026-02-20T22:40:24.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 438971392 bytes 2026-02-20T22:40:24.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 493678592 bytes 2026-02-20T22:40:24.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 539731968 bytes 2026-02-20T22:40:24.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 540346368 bytes 2026-02-20T22:40:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 602220544 bytes 2026-02-20T22:40:24.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 655807488 bytes 2026-02-20T22:40:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 738465792 bytes 2026-02-20T22:40:24.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 765759488 bytes 2026-02-20T22:40:25.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 786529280 bytes 2026-02-20T22:40:25.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 794598400 bytes 2026-02-20T22:40:25.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 861977600 bytes 2026-02-20T22:40:25.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 914494464 bytes 2026-02-20T22:40:25.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 962831360 bytes 2026-02-20T22:40:25.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1027066880 bytes 2026-02-20T22:40:25.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1090878464 bytes 2026-02-20T22:40:25.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1091506176 bytes 2026-02-20T22:40:25.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1157980160 bytes 2026-02-20T22:40:25.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1236237312 bytes 2026-02-20T22:40:25.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1285489664 bytes 2026-02-20T22:40:25.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1319208960 bytes 2026-02-20T22:40:25.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1401932800 bytes 2026-02-20T22:40:25.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1430904832 bytes 2026-02-20T22:40:25.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1457923072 bytes 2026-02-20T22:40:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1592756224 bytes 2026-02-20T22:40:25.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1612169216 bytes 2026-02-20T22:40:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1639332864 bytes 2026-02-20T22:40:26.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1685928960 bytes 2026-02-20T22:40:26.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1735564288 bytes 2026-02-20T22:40:26.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1791404032 bytes 2026-02-20T22:40:26.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1846274048 bytes 2026-02-20T22:40:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1903380480 bytes 2026-02-20T22:40:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1960796160 bytes 2026-02-20T22:40:26.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1991705600 bytes 2026-02-20T22:40:26.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2030742528 bytes 2026-02-20T22:40:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2036443136 bytes 2026-02-20T22:40:26.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2058889216 bytes 2026-02-20T22:40:26.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2085120000 bytes 2026-02-20T22:40:26.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2117990400 bytes 2026-02-20T22:40:26.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2139645952 bytes 2026-02-20T22:40:26.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2159991808 bytes 2026-02-20T22:40:26.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2185684992 bytes 2026-02-20T22:40:26.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2241449984 bytes 2026-02-20T22:40:26.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2267658240 bytes 2026-02-20T22:40:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2284587008 bytes 2026-02-20T22:40:27.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2371971072 bytes 2026-02-20T22:40:27.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2404507648 bytes 2026-02-20T22:40:27.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2471244800 bytes 2026-02-20T22:40:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2498201600 bytes 2026-02-20T22:40:27.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2533629952 bytes 2026-02-20T22:40:27.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2616252416 bytes 2026-02-20T22:40:27.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2623723520 bytes 2026-02-20T22:40:27.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2683115520 bytes 2026-02-20T22:40:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2707735552 bytes 2026-02-20T22:40:27.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2713083904 bytes 2026-02-20T22:40:27.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2778470400 bytes 2026-02-20T22:40:27.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2845480960 bytes 2026-02-20T22:40:27.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2870558720 bytes 2026-02-20T22:40:27.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2930634752 bytes 2026-02-20T22:40:27.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2980298752 bytes 2026-02-20T22:40:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3042977792 bytes 2026-02-20T22:40:27.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3062523904 bytes 2026-02-20T22:40:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3063140352 bytes 2026-02-20T22:40:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3125516288 bytes 2026-02-20T22:40:28.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3212869632 bytes 2026-02-20T22:40:28.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3226460160 bytes 2026-02-20T22:40:28.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3298987008 bytes 2026-02-20T22:40:28.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3353515008 bytes 2026-02-20T22:40:28.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3381468160 bytes 2026-02-20T22:40:28.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3403376640 bytes 2026-02-20T22:40:28.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3442221056 bytes 2026-02-20T22:40:28.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3455673344 bytes 2026-02-20T22:40:28.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3531868160 bytes 2026-02-20T22:40:28.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3597003776 bytes 2026-02-20T22:40:28.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3664866304 bytes 2026-02-20T22:40:28.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3734514688 bytes 2026-02-20T22:40:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3744412672 bytes 2026-02-20T22:40:28.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3779953664 bytes 2026-02-20T22:40:29.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3841242112 bytes 2026-02-20T22:40:29.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3899057152 bytes 2026-02-20T22:40:29.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4002037760 bytes 2026-02-20T22:40:29.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4023958528 bytes 2026-02-20T22:40:29.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4087671808 bytes 2026-02-20T22:40:29.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4208949248 bytes 2026-02-20T22:40:29.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4283402240 bytes 2026-02-20T22:40:29.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4334443520 bytes 2026-02-20T22:40:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4412173312 bytes 2026-02-20T22:40:29.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4493209600 bytes 2026-02-20T22:40:29.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4603734016 bytes 2026-02-20T22:40:29.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4643559424 bytes 2026-02-20T22:40:29.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4667782144 bytes 2026-02-20T22:40:29.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_28 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:29.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_28 128M 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:40:29.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:29.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2026-02-20T22:40:30.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:30.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2026-02-20T22:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_28': 2026-02-20T22:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:30.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13ae48f2415c 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13ae48f2415c 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:29 2026 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:29 2026 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:29 2026 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:30.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: f3e3b67e-63fc-4f0c-a1ad-75bec4cf631a 2026-02-20T22:40:30.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_28 2026-02-20T22:40:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:30.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:40:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:40:30.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:30.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:30.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 34469888 bytes 2026-02-20T22:40:31.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 100507648 bytes 2026-02-20T22:40:31.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 117694464 bytes 2026-02-20T22:40:31.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 175538176 bytes 2026-02-20T22:40:31.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 245591040 bytes 2026-02-20T22:40:31.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 258426880 bytes 2026-02-20T22:40:31.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 318490624 bytes 2026-02-20T22:40:31.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 326337536 bytes 2026-02-20T22:40:31.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 419118080 bytes 2026-02-20T22:40:31.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 487302144 bytes 2026-02-20T22:40:31.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 522657792 bytes 2026-02-20T22:40:31.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 565407744 bytes 2026-02-20T22:40:31.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 698861568 bytes 2026-02-20T22:40:31.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 762883072 bytes 2026-02-20T22:40:31.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 780840960 bytes 2026-02-20T22:40:31.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 849832960 bytes 2026-02-20T22:40:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 891834368 bytes 2026-02-20T22:40:32.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 930023424 bytes 2026-02-20T22:40:32.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 942275584 bytes 2026-02-20T22:40:32.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1029476352 bytes 2026-02-20T22:40:32.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1070430208 bytes 2026-02-20T22:40:32.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1111917568 bytes 2026-02-20T22:40:32.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1160848384 bytes 2026-02-20T22:40:32.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1189612544 bytes 2026-02-20T22:40:32.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1214535680 bytes 2026-02-20T22:40:32.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1305254912 bytes 2026-02-20T22:40:32.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1345354752 bytes 2026-02-20T22:40:32.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1390432256 bytes 2026-02-20T22:40:32.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1400592384 bytes 2026-02-20T22:40:32.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1457925120 bytes 2026-02-20T22:40:32.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1474489344 bytes 2026-02-20T22:40:32.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1515973632 bytes 2026-02-20T22:40:32.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1573928960 bytes 2026-02-20T22:40:32.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1614590976 bytes 2026-02-20T22:40:33.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1696295936 bytes 2026-02-20T22:40:33.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1746678784 bytes 2026-02-20T22:40:33.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1811227648 bytes 2026-02-20T22:40:33.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1859542016 bytes 2026-02-20T22:40:33.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1874498560 bytes 2026-02-20T22:40:33.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1934054400 bytes 2026-02-20T22:40:33.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1957610496 bytes 2026-02-20T22:40:33.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1960952832 bytes 2026-02-20T22:40:33.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1988871168 bytes 2026-02-20T22:40:33.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2047015936 bytes 2026-02-20T22:40:33.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2077962240 bytes 2026-02-20T22:40:33.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2117272576 bytes 2026-02-20T22:40:33.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2189174784 bytes 2026-02-20T22:40:33.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2217814016 bytes 2026-02-20T22:40:33.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2256208896 bytes 2026-02-20T22:40:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2321389568 bytes 2026-02-20T22:40:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2377013248 bytes 2026-02-20T22:40:34.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2426231808 bytes 2026-02-20T22:40:34.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2494237696 bytes 2026-02-20T22:40:34.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2536754176 bytes 2026-02-20T22:40:34.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2562373632 bytes 2026-02-20T22:40:34.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2586480640 bytes 2026-02-20T22:40:34.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2652134400 bytes 2026-02-20T22:40:34.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2660069376 bytes 2026-02-20T22:40:34.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2678935552 bytes 2026-02-20T22:40:34.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2738135040 bytes 2026-02-20T22:40:34.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2786718720 bytes 2026-02-20T22:40:34.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2824279040 bytes 2026-02-20T22:40:34.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2865908736 bytes 2026-02-20T22:40:34.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2916016128 bytes 2026-02-20T22:40:34.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3016526848 bytes 2026-02-20T22:40:34.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3093031936 bytes 2026-02-20T22:40:34.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3123113984 bytes 2026-02-20T22:40:34.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3204686848 bytes 2026-02-20T22:40:35.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3229095936 bytes 2026-02-20T22:40:35.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3256341504 bytes 2026-02-20T22:40:35.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3286440960 bytes 2026-02-20T22:40:35.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3287072768 bytes 2026-02-20T22:40:35.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3328211968 bytes 2026-02-20T22:40:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3358685184 bytes 2026-02-20T22:40:35.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3362428928 bytes 2026-02-20T22:40:35.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3372075008 bytes 2026-02-20T22:40:35.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3506094080 bytes 2026-02-20T22:40:35.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3638451200 bytes 2026-02-20T22:40:35.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3692806144 bytes 2026-02-20T22:40:35.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3704948736 bytes 2026-02-20T22:40:35.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3720197120 bytes 2026-02-20T22:40:35.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3771138048 bytes 2026-02-20T22:40:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3832619008 bytes 2026-02-20T22:40:36.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3874520064 bytes 2026-02-20T22:40:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3971913728 bytes 2026-02-20T22:40:36.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3986432000 bytes 2026-02-20T22:40:36.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3994226688 bytes 2026-02-20T22:40:36.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4034361344 bytes 2026-02-20T22:40:36.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4034967552 bytes 2026-02-20T22:40:36.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4035593216 bytes 2026-02-20T22:40:36.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4054382592 bytes 2026-02-20T22:40:36.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4073376768 bytes 2026-02-20T22:40:36.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4116737024 bytes 2026-02-20T22:40:36.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4132765696 bytes 2026-02-20T22:40:36.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4159068160 bytes 2026-02-20T22:40:36.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4269367296 bytes 2026-02-20T22:40:36.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4346253312 bytes 2026-02-20T22:40:36.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4376512512 bytes 2026-02-20T22:40:36.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4449613824 bytes 2026-02-20T22:40:36.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4503356416 bytes 2026-02-20T22:40:37.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4533233664 bytes 2026-02-20T22:40:37.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4593022976 bytes 2026-02-20T22:40:37.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4629230592 bytes 2026-02-20T22:40:37.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4713117696 bytes 2026-02-20T22:40:37.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4803880960 bytes 2026-02-20T22:40:37.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4805720064 bytes 2026-02-20T22:40:37.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4814389248 bytes 2026-02-20T22:40:37.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4836470784 bytes 2026-02-20T22:40:37.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4855205888 bytes 2026-02-20T22:40:37.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4933851136 bytes 2026-02-20T22:40:37.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4980129792 bytes 2026-02-20T22:40:37.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4980754432 bytes 2026-02-20T22:40:37.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5007836160 bytes 2026-02-20T22:40:37.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5028036608 bytes 2026-02-20T22:40:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5038873600 bytes 2026-02-20T22:40:37.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5093453824 bytes 2026-02-20T22:40:37.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5096930304 bytes 2026-02-20T22:40:37.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5122632704 bytes 2026-02-20T22:40:38.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5202744320 bytes 2026-02-20T22:40:38.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5228759040 bytes 2026-02-20T22:40:38.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5229375488 bytes 2026-02-20T22:40:38.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5251590144 bytes 2026-02-20T22:40:38.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5274608640 bytes 2026-02-20T22:40:38.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5289828352 bytes 2026-02-20T22:40:38.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5334272000 bytes 2026-02-20T22:40:38.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5416866816 bytes 2026-02-20T22:40:38.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5435260928 bytes 2026-02-20T22:40:38.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5435865088 bytes 2026-02-20T22:40:38.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5483104256 bytes 2026-02-20T22:40:38.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5491699712 bytes 2026-02-20T22:40:38.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5539728384 bytes 2026-02-20T22:40:38.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5557197824 bytes 2026-02-20T22:40:38.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5613562880 bytes 2026-02-20T22:40:38.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5616672768 bytes 2026-02-20T22:40:38.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5642024960 bytes 2026-02-20T22:40:38.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5678863360 bytes 2026-02-20T22:40:38.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5717445632 bytes 2026-02-20T22:40:38.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5770818560 bytes 2026-02-20T22:40:39.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5805484032 bytes 2026-02-20T22:40:39.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5824351232 bytes 2026-02-20T22:40:39.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5856849920 bytes 2026-02-20T22:40:39.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5909168128 bytes 2026-02-20T22:40:39.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5944272896 bytes 2026-02-20T22:40:39.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6046529536 bytes 2026-02-20T22:40:39.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6072247296 bytes 2026-02-20T22:40:39.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6143718400 bytes 2026-02-20T22:40:39.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6179391488 bytes 2026-02-20T22:40:39.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6193611776 bytes 2026-02-20T22:40:39.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6201010176 bytes 2026-02-20T22:40:39.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6244152320 bytes 2026-02-20T22:40:39.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6257112064 bytes 2026-02-20T22:40:39.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6281654272 bytes 2026-02-20T22:40:39.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6298431488 bytes 2026-02-20T22:40:39.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6358778880 bytes 2026-02-20T22:40:39.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6399019008 bytes 2026-02-20T22:40:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6453226496 bytes 2026-02-20T22:40:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6486206464 bytes 2026-02-20T22:40:40.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6512631808 bytes 2026-02-20T22:40:40.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6632749056 bytes 2026-02-20T22:40:40.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6677049344 bytes 2026-02-20T22:40:40.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6774749184 bytes 2026-02-20T22:40:40.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6813947904 bytes 2026-02-20T22:40:40.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6823917568 bytes 2026-02-20T22:40:40.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6838700032 bytes 2026-02-20T22:40:40.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6896850944 bytes 2026-02-20T22:40:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6929497088 bytes 2026-02-20T22:40:40.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6958032896 bytes 2026-02-20T22:40:40.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7020172288 bytes 2026-02-20T22:40:40.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7051639808 bytes 2026-02-20T22:40:40.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7141757952 bytes 2026-02-20T22:40:40.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7217518592 bytes 2026-02-20T22:40:40.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7248734208 bytes 2026-02-20T22:40:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7273228288 bytes 2026-02-20T22:40:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7341315072 bytes 2026-02-20T22:40:41.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7341945856 bytes 2026-02-20T22:40:41.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7383096320 bytes 2026-02-20T22:40:41.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7420688384 bytes 2026-02-20T22:40:41.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7477134336 bytes 2026-02-20T22:40:41.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7541151744 bytes 2026-02-20T22:40:41.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7576738816 bytes 2026-02-20T22:40:41.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7643745280 bytes 2026-02-20T22:40:41.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7660218368 bytes 2026-02-20T22:40:41.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7686368256 bytes 2026-02-20T22:40:41.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7754490880 bytes 2026-02-20T22:40:41.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7770382336 bytes 2026-02-20T22:40:41.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7798398976 bytes 2026-02-20T22:40:41.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7816731648 bytes 2026-02-20T22:40:41.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7838157824 bytes 2026-02-20T22:40:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7905883136 bytes 2026-02-20T22:40:41.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7911567360 bytes 2026-02-20T22:40:41.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8003156992 bytes 2026-02-20T22:40:42.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8021386240 bytes 2026-02-20T22:40:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8074205184 bytes 2026-02-20T22:40:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8125713408 bytes 2026-02-20T22:40:42.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8166575104 bytes 2026-02-20T22:40:42.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8213287936 bytes 2026-02-20T22:40:42.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8260354048 bytes 2026-02-20T22:40:42.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8292916224 bytes 2026-02-20T22:40:42.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8313854976 bytes 2026-02-20T22:40:42.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8322577408 bytes 2026-02-20T22:40:42.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8336423936 bytes 2026-02-20T22:40:42.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8378203136 bytes 2026-02-20T22:40:42.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8433483776 bytes 2026-02-20T22:40:42.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8470376448 bytes 2026-02-20T22:40:42.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8535139328 bytes 2026-02-20T22:40:42.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8620320768 bytes 2026-02-20T22:40:42.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8649195520 bytes 2026-02-20T22:40:42.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8806896640 bytes 2026-02-20T22:40:42.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8812581888 bytes 2026-02-20T22:40:43.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8901639168 bytes 2026-02-20T22:40:43.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8926976000 bytes 2026-02-20T22:40:43.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 8972146688 bytes 2026-02-20T22:40:43.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9102905344 bytes 2026-02-20T22:40:43.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9140450304 bytes 2026-02-20T22:40:43.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9208793088 bytes 2026-02-20T22:40:43.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9228848128 bytes 2026-02-20T22:40:43.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9325405184 bytes 2026-02-20T22:40:43.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9368812544 bytes 2026-02-20T22:40:43.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9399688192 bytes 2026-02-20T22:40:43.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9446044672 bytes 2026-02-20T22:40:43.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9474393088 bytes 2026-02-20T22:40:43.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9498168320 bytes 2026-02-20T22:40:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9498781696 bytes 2026-02-20T22:40:43.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9542352896 bytes 2026-02-20T22:40:43.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:43.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_29 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:40:43.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_29 128M 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:43.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:43.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2026-02-20T22:40:43.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:43.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2026-02-20T22:40:44.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:44.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2026-02-20T22:40:44.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_29': 2026-02-20T22:40:44.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:44.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:44.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13b676f414da 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13b676f414da 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:43 2026 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:43 2026 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:43 2026 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:44.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:44.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: b44f2e80-554c-46e7-9cff-22a5d35c7769 2026-02-20T22:40:44.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_29 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:40:44.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:40:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:44.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 61673472 bytes 2026-02-20T22:40:45.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 118223872 bytes 2026-02-20T22:40:45.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 124648448 bytes 2026-02-20T22:40:45.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 136796160 bytes 2026-02-20T22:40:45.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 150680576 bytes 2026-02-20T22:40:45.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 242449408 bytes 2026-02-20T22:40:45.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 274680832 bytes 2026-02-20T22:40:45.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 278736896 bytes 2026-02-20T22:40:45.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 298394624 bytes 2026-02-20T22:40:45.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 333838336 bytes 2026-02-20T22:40:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 364453888 bytes 2026-02-20T22:40:45.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 410468352 bytes 2026-02-20T22:40:45.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 462346240 bytes 2026-02-20T22:40:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 478390272 bytes 2026-02-20T22:40:45.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 529248256 bytes 2026-02-20T22:40:45.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 608772096 bytes 2026-02-20T22:40:45.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 615989248 bytes 2026-02-20T22:40:45.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 617839616 bytes 2026-02-20T22:40:45.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 643249152 bytes 2026-02-20T22:40:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 677226496 bytes 2026-02-20T22:40:46.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 797429760 bytes 2026-02-20T22:40:46.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 835553280 bytes 2026-02-20T22:40:46.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 892196864 bytes 2026-02-20T22:40:46.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 939549696 bytes 2026-02-20T22:40:46.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 991408128 bytes 2026-02-20T22:40:46.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1021080576 bytes 2026-02-20T22:40:46.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1073464320 bytes 2026-02-20T22:40:46.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1088777216 bytes 2026-02-20T22:40:46.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1151550464 bytes 2026-02-20T22:40:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1211552768 bytes 2026-02-20T22:40:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1253846016 bytes 2026-02-20T22:40:46.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1264555008 bytes 2026-02-20T22:40:46.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1353035776 bytes 2026-02-20T22:40:46.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1389544448 bytes 2026-02-20T22:40:46.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1417226240 bytes 2026-02-20T22:40:46.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1491588096 bytes 2026-02-20T22:40:47.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1528300544 bytes 2026-02-20T22:40:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1573070848 bytes 2026-02-20T22:40:47.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1594316800 bytes 2026-02-20T22:40:47.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1623368704 bytes 2026-02-20T22:40:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1657396224 bytes 2026-02-20T22:40:47.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1707044864 bytes 2026-02-20T22:40:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1715454976 bytes 2026-02-20T22:40:47.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1773063168 bytes 2026-02-20T22:40:47.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1776380928 bytes 2026-02-20T22:40:47.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1838020608 bytes 2026-02-20T22:40:47.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1881883648 bytes 2026-02-20T22:40:47.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1908674560 bytes 2026-02-20T22:40:47.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1953031168 bytes 2026-02-20T22:40:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1962876928 bytes 2026-02-20T22:40:47.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2036078592 bytes 2026-02-20T22:40:47.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2137804800 bytes 2026-02-20T22:40:47.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2182758400 bytes 2026-02-20T22:40:47.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2226256896 bytes 2026-02-20T22:40:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2263002112 bytes 2026-02-20T22:40:48.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2315362304 bytes 2026-02-20T22:40:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2330260480 bytes 2026-02-20T22:40:48.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2361611264 bytes 2026-02-20T22:40:48.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2373782528 bytes 2026-02-20T22:40:48.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2457181184 bytes 2026-02-20T22:40:48.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2494626816 bytes 2026-02-20T22:40:48.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2527361024 bytes 2026-02-20T22:40:48.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2568181760 bytes 2026-02-20T22:40:48.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2620350464 bytes 2026-02-20T22:40:48.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2677395456 bytes 2026-02-20T22:40:48.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2713088000 bytes 2026-02-20T22:40:48.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2751600640 bytes 2026-02-20T22:40:48.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2787670016 bytes 2026-02-20T22:40:48.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2799728640 bytes 2026-02-20T22:40:48.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2810762240 bytes 2026-02-20T22:40:48.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2836690944 bytes 2026-02-20T22:40:48.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2854619136 bytes 2026-02-20T22:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2929511424 bytes 2026-02-20T22:40:49.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2962810880 bytes 2026-02-20T22:40:49.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3056877568 bytes 2026-02-20T22:40:49.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3080878080 bytes 2026-02-20T22:40:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3095110656 bytes 2026-02-20T22:40:49.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3144308736 bytes 2026-02-20T22:40:49.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3183713280 bytes 2026-02-20T22:40:49.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3272239104 bytes 2026-02-20T22:40:49.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3308043264 bytes 2026-02-20T22:40:49.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3421467648 bytes 2026-02-20T22:40:49.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3427843072 bytes 2026-02-20T22:40:49.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3434700800 bytes 2026-02-20T22:40:49.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3458872320 bytes 2026-02-20T22:40:49.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3477227520 bytes 2026-02-20T22:40:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3533306880 bytes 2026-02-20T22:40:50.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3620038656 bytes 2026-02-20T22:40:50.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3652620288 bytes 2026-02-20T22:40:50.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3690455040 bytes 2026-02-20T22:40:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3723955200 bytes 2026-02-20T22:40:50.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3742419968 bytes 2026-02-20T22:40:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3743011840 bytes 2026-02-20T22:40:50.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3782987776 bytes 2026-02-20T22:40:50.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3812165632 bytes 2026-02-20T22:40:50.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3877782528 bytes 2026-02-20T22:40:50.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3966086144 bytes 2026-02-20T22:40:50.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4003422208 bytes 2026-02-20T22:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4058039296 bytes 2026-02-20T22:40:50.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4112614400 bytes 2026-02-20T22:40:50.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4127719424 bytes 2026-02-20T22:40:50.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4166737920 bytes 2026-02-20T22:40:50.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4238512128 bytes 2026-02-20T22:40:51.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4302216192 bytes 2026-02-20T22:40:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4345470976 bytes 2026-02-20T22:40:51.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4398235648 bytes 2026-02-20T22:40:51.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4451051520 bytes 2026-02-20T22:40:51.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4465218560 bytes 2026-02-20T22:40:51.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4508737536 bytes 2026-02-20T22:40:51.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4553785344 bytes 2026-02-20T22:40:51.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4617674752 bytes 2026-02-20T22:40:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4659721216 bytes 2026-02-20T22:40:51.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4700323840 bytes 2026-02-20T22:40:51.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4720073728 bytes 2026-02-20T22:40:51.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4776217600 bytes 2026-02-20T22:40:51.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4776828928 bytes 2026-02-20T22:40:51.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4833020928 bytes 2026-02-20T22:40:51.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4883574784 bytes 2026-02-20T22:40:51.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4902782976 bytes 2026-02-20T22:40:51.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4952865792 bytes 2026-02-20T22:40:51.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_30 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_30 128M 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:40:51.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:51.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2026-02-20T22:40:51.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:51.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2026-02-20T22:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:51.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2026-02-20T22:40:52.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:40:52.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_30': 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13be2ea79d30 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13be2ea79d30 2026-02-20T22:40:52.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:51 2026 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:51 2026 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:51 2026 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 76b57b0a-af4f-4c2a-9a97-d9e658cc43f5 2026-02-20T22:40:52.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_30 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:40:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:40:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:40:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:40:52.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:40:53.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 9005056 bytes 2026-02-20T22:40:53.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 77702144 bytes 2026-02-20T22:40:53.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 127330304 bytes 2026-02-20T22:40:53.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 145976320 bytes 2026-02-20T22:40:53.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 190201856 bytes 2026-02-20T22:40:53.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 246089728 bytes 2026-02-20T22:40:53.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 261497856 bytes 2026-02-20T22:40:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 324337664 bytes 2026-02-20T22:40:53.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 375500800 bytes 2026-02-20T22:40:53.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 389704704 bytes 2026-02-20T22:40:53.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 394917888 bytes 2026-02-20T22:40:53.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 414217216 bytes 2026-02-20T22:40:53.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 469925888 bytes 2026-02-20T22:40:53.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 486584320 bytes 2026-02-20T22:40:53.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 511697920 bytes 2026-02-20T22:40:53.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 545090560 bytes 2026-02-20T22:40:53.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 653748224 bytes 2026-02-20T22:40:53.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 700840960 bytes 2026-02-20T22:40:53.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 746548224 bytes 2026-02-20T22:40:54.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 784400384 bytes 2026-02-20T22:40:54.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 825417728 bytes 2026-02-20T22:40:54.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 865562624 bytes 2026-02-20T22:40:54.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 883340288 bytes 2026-02-20T22:40:54.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 940635136 bytes 2026-02-20T22:40:54.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 943662080 bytes 2026-02-20T22:40:54.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 944293888 bytes 2026-02-20T22:40:54.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1009768448 bytes 2026-02-20T22:40:54.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1041675264 bytes 2026-02-20T22:40:54.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1084241920 bytes 2026-02-20T22:40:54.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1110709248 bytes 2026-02-20T22:40:54.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1136892928 bytes 2026-02-20T22:40:54.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1145326592 bytes 2026-02-20T22:40:54.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1194659840 bytes 2026-02-20T22:40:54.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1251745792 bytes 2026-02-20T22:40:54.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1276304384 bytes 2026-02-20T22:40:54.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1292132352 bytes 2026-02-20T22:40:54.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1321464832 bytes 2026-02-20T22:40:54.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1353938944 bytes 2026-02-20T22:40:54.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1354537984 bytes 2026-02-20T22:40:54.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1399093248 bytes 2026-02-20T22:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1413696512 bytes 2026-02-20T22:40:55.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1458574336 bytes 2026-02-20T22:40:55.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1504360448 bytes 2026-02-20T22:40:55.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1519238144 bytes 2026-02-20T22:40:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1564731392 bytes 2026-02-20T22:40:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1589432320 bytes 2026-02-20T22:40:55.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1692368896 bytes 2026-02-20T22:40:55.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1730872320 bytes 2026-02-20T22:40:55.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1778170880 bytes 2026-02-20T22:40:55.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1802470400 bytes 2026-02-20T22:40:55.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1840904192 bytes 2026-02-20T22:40:55.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1897832448 bytes 2026-02-20T22:40:55.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 1904567296 bytes 2026-02-20T22:40:55.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 1934953472 bytes 2026-02-20T22:40:55.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 1983457280 bytes 2026-02-20T22:40:55.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2007532544 bytes 2026-02-20T22:40:55.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2041936896 bytes 2026-02-20T22:40:55.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2132608000 bytes 2026-02-20T22:40:56.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2173369344 bytes 2026-02-20T22:40:56.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2225094656 bytes 2026-02-20T22:40:56.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2313310208 bytes 2026-02-20T22:40:56.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2371598336 bytes 2026-02-20T22:40:56.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2459893760 bytes 2026-02-20T22:40:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2546001920 bytes 2026-02-20T22:40:56.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2546619392 bytes 2026-02-20T22:40:56.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2573843456 bytes 2026-02-20T22:40:56.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2609356800 bytes 2026-02-20T22:40:56.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2664543232 bytes 2026-02-20T22:40:56.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2688998400 bytes 2026-02-20T22:40:56.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2726396928 bytes 2026-02-20T22:40:56.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2740194304 bytes 2026-02-20T22:40:56.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2814067712 bytes 2026-02-20T22:40:56.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2821627904 bytes 2026-02-20T22:40:56.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2822280192 bytes 2026-02-20T22:40:56.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2844357632 bytes 2026-02-20T22:40:56.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 2875048960 bytes 2026-02-20T22:40:56.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 2875653120 bytes 2026-02-20T22:40:57.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 2894768128 bytes 2026-02-20T22:40:57.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 2955288576 bytes 2026-02-20T22:40:57.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3020513280 bytes 2026-02-20T22:40:57.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3054895104 bytes 2026-02-20T22:40:57.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3087587328 bytes 2026-02-20T22:40:57.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3118036992 bytes 2026-02-20T22:40:57.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3171116032 bytes 2026-02-20T22:40:57.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3199367168 bytes 2026-02-20T22:40:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3199978496 bytes 2026-02-20T22:40:57.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3229399040 bytes 2026-02-20T22:40:57.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3284683776 bytes 2026-02-20T22:40:57.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3300853760 bytes 2026-02-20T22:40:57.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3363268608 bytes 2026-02-20T22:40:57.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3399255040 bytes 2026-02-20T22:40:57.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3456464896 bytes 2026-02-20T22:40:57.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3481166848 bytes 2026-02-20T22:40:57.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3493489664 bytes 2026-02-20T22:40:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3501145088 bytes 2026-02-20T22:40:57.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3501745152 bytes 2026-02-20T22:40:58.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3543147520 bytes 2026-02-20T22:40:58.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3591968768 bytes 2026-02-20T22:40:58.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3651958784 bytes 2026-02-20T22:40:58.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3676879872 bytes 2026-02-20T22:40:58.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 3720067072 bytes 2026-02-20T22:40:58.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 3735046144 bytes 2026-02-20T22:40:58.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 3781064704 bytes 2026-02-20T22:40:58.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 3857018880 bytes 2026-02-20T22:40:58.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 3919095808 bytes 2026-02-20T22:40:58.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 3988909056 bytes 2026-02-20T22:40:58.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4060626944 bytes 2026-02-20T22:40:58.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4139242496 bytes 2026-02-20T22:40:58.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4223706112 bytes 2026-02-20T22:40:58.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4263823360 bytes 2026-02-20T22:40:58.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4344871936 bytes 2026-02-20T22:40:58.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4351021056 bytes 2026-02-20T22:40:59.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4368241664 bytes 2026-02-20T22:40:59.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4444459008 bytes 2026-02-20T22:40:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4485148672 bytes 2026-02-20T22:40:59.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4499579904 bytes 2026-02-20T22:40:59.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4526071808 bytes 2026-02-20T22:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4543374336 bytes 2026-02-20T22:40:59.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4581792768 bytes 2026-02-20T22:40:59.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4632164352 bytes 2026-02-20T22:40:59.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4692724736 bytes 2026-02-20T22:40:59.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4715237376 bytes 2026-02-20T22:40:59.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4753082368 bytes 2026-02-20T22:40:59.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4775164928 bytes 2026-02-20T22:40:59.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4805715968 bytes 2026-02-20T22:40:59.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 4819670016 bytes 2026-02-20T22:40:59.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 4855035904 bytes 2026-02-20T22:40:59.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 4894609408 bytes 2026-02-20T22:40:59.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 4930631680 bytes 2026-02-20T22:40:59.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 4955084800 bytes 2026-02-20T22:40:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 4984217600 bytes 2026-02-20T22:40:59.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:40:59.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:40:59.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_31 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:59.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_31 128M 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:40:59.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:40:59.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2026-02-20T22:40:59.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:40:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:40:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:40:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:40:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2026-02-20T22:41:00.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:00.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2026-02-20T22:41:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_31': 2026-02-20T22:41:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:00.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13c7f7e5a9f1 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13c7f7e5a9f1 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:59 2026 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:59 2026 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:59 2026 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:00.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 0d6df238-c085-4a6e-ad2d-7aa25e3b6e40 2026-02-20T22:41:00.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_31 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:41:00.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:41:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:00.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:01.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 64811008 bytes 2026-02-20T22:41:01.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 173642752 bytes 2026-02-20T22:41:01.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 217990144 bytes 2026-02-20T22:41:01.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 253861888 bytes 2026-02-20T22:41:01.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 279025664 bytes 2026-02-20T22:41:01.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 317287424 bytes 2026-02-20T22:41:01.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 340615168 bytes 2026-02-20T22:41:01.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 397541376 bytes 2026-02-20T22:41:01.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4265746954317393) > weight (0.25) 2026-02-20T22:41:01.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.46196141141157987) > weight (0.25) 2026-02-20T22:41:01.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.26703177490772756) > weight (0.25) 2026-02-20T22:41:01.520 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6047900959401954) > weight (0.25) 2026-02-20T22:41:01.520 INFO:tasks.rbd_mirror_thrash:waiting for 71.75245540178926 secs before thrashing 2026-02-20T22:41:01.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 398176256 bytes 2026-02-20T22:41:01.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 423778304 bytes 2026-02-20T22:41:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 477108224 bytes 2026-02-20T22:41:01.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 499536896 bytes 2026-02-20T22:41:01.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 512787456 bytes 2026-02-20T22:41:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 582246400 bytes 2026-02-20T22:41:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 644890624 bytes 2026-02-20T22:41:01.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 667000832 bytes 2026-02-20T22:41:02.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 776540160 bytes 2026-02-20T22:41:02.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 797895680 bytes 2026-02-20T22:41:02.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 883635200 bytes 2026-02-20T22:41:02.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 902106112 bytes 2026-02-20T22:41:02.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 926854144 bytes 2026-02-20T22:41:02.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 970211328 bytes 2026-02-20T22:41:02.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 993071104 bytes 2026-02-20T22:41:02.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 997985280 bytes 2026-02-20T22:41:02.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1042674688 bytes 2026-02-20T22:41:02.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1086519296 bytes 2026-02-20T22:41:02.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1143121920 bytes 2026-02-20T22:41:02.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1230601216 bytes 2026-02-20T22:41:02.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1288710144 bytes 2026-02-20T22:41:02.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1329166336 bytes 2026-02-20T22:41:02.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1393347584 bytes 2026-02-20T22:41:02.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1440249856 bytes 2026-02-20T22:41:03.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1505908736 bytes 2026-02-20T22:41:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1565451264 bytes 2026-02-20T22:41:03.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1623243776 bytes 2026-02-20T22:41:03.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1670914048 bytes 2026-02-20T22:41:03.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1724166144 bytes 2026-02-20T22:41:03.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1791606784 bytes 2026-02-20T22:41:03.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1840107520 bytes 2026-02-20T22:41:03.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1875079168 bytes 2026-02-20T22:41:03.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1895391232 bytes 2026-02-20T22:41:03.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1926836224 bytes 2026-02-20T22:41:03.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2034779136 bytes 2026-02-20T22:41:03.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2064171008 bytes 2026-02-20T22:41:03.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2083250176 bytes 2026-02-20T22:41:03.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2132631552 bytes 2026-02-20T22:41:03.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2173117440 bytes 2026-02-20T22:41:03.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2196554752 bytes 2026-02-20T22:41:03.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2250711040 bytes 2026-02-20T22:41:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2280809472 bytes 2026-02-20T22:41:04.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2334715904 bytes 2026-02-20T22:41:04.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2370450432 bytes 2026-02-20T22:41:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2410593280 bytes 2026-02-20T22:41:04.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2425333760 bytes 2026-02-20T22:41:04.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2477144064 bytes 2026-02-20T22:41:04.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2511223808 bytes 2026-02-20T22:41:04.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2545391616 bytes 2026-02-20T22:41:04.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2603696128 bytes 2026-02-20T22:41:04.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2622993408 bytes 2026-02-20T22:41:04.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2673614848 bytes 2026-02-20T22:41:04.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2705814528 bytes 2026-02-20T22:41:04.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2770632704 bytes 2026-02-20T22:41:04.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2829734912 bytes 2026-02-20T22:41:04.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2839244800 bytes 2026-02-20T22:41:04.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2867154944 bytes 2026-02-20T22:41:04.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2935152640 bytes 2026-02-20T22:41:05.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2988190720 bytes 2026-02-20T22:41:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3118195712 bytes 2026-02-20T22:41:05.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3204499456 bytes 2026-02-20T22:41:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3235865600 bytes 2026-02-20T22:41:05.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3267900416 bytes 2026-02-20T22:41:05.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3373590528 bytes 2026-02-20T22:41:05.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3445516288 bytes 2026-02-20T22:41:05.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3448542208 bytes 2026-02-20T22:41:05.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3460733952 bytes 2026-02-20T22:41:05.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3479906304 bytes 2026-02-20T22:41:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3539222528 bytes 2026-02-20T22:41:05.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3555947520 bytes 2026-02-20T22:41:05.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3605668864 bytes 2026-02-20T22:41:05.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3648075776 bytes 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_32 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_32 128M 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:05.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2026-02-20T22:41:06.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:06.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2026-02-20T22:41:06.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_32': 2026-02-20T22:41:06.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:06.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:06.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13d0edb7eb15 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13d0edb7eb15 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:05 2026 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:05 2026 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:05 2026 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3e048625-939b-485e-ae8c-d4b0fec7ae1d 2026-02-20T22:41:06.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_32 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:41:06.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:06.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:41:06.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:06.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:06.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 41936896 bytes 2026-02-20T22:41:06.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 58416128 bytes 2026-02-20T22:41:06.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 59018240 bytes 2026-02-20T22:41:06.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 108041216 bytes 2026-02-20T22:41:06.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 151885824 bytes 2026-02-20T22:41:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 217412608 bytes 2026-02-20T22:41:07.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 225061888 bytes 2026-02-20T22:41:07.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 318478336 bytes 2026-02-20T22:41:07.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 380046336 bytes 2026-02-20T22:41:07.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 448620544 bytes 2026-02-20T22:41:07.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 502202368 bytes 2026-02-20T22:41:07.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 563681280 bytes 2026-02-20T22:41:07.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 619139072 bytes 2026-02-20T22:41:07.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 715012096 bytes 2026-02-20T22:41:07.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 715644928 bytes 2026-02-20T22:41:07.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 785666048 bytes 2026-02-20T22:41:07.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 802590720 bytes 2026-02-20T22:41:07.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 832668672 bytes 2026-02-20T22:41:07.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 884887552 bytes 2026-02-20T22:41:07.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 898647040 bytes 2026-02-20T22:41:07.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 949675008 bytes 2026-02-20T22:41:08.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1007072256 bytes 2026-02-20T22:41:08.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1050385408 bytes 2026-02-20T22:41:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1093299200 bytes 2026-02-20T22:41:08.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1160424448 bytes 2026-02-20T22:41:08.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1185698816 bytes 2026-02-20T22:41:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1241549824 bytes 2026-02-20T22:41:08.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1364395008 bytes 2026-02-20T22:41:08.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1419581440 bytes 2026-02-20T22:41:08.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1440668672 bytes 2026-02-20T22:41:08.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1469960192 bytes 2026-02-20T22:41:08.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1509213184 bytes 2026-02-20T22:41:08.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1549392896 bytes 2026-02-20T22:41:08.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1586954240 bytes 2026-02-20T22:41:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1625446400 bytes 2026-02-20T22:41:08.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1672666112 bytes 2026-02-20T22:41:08.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1729031168 bytes 2026-02-20T22:41:09.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1735072768 bytes 2026-02-20T22:41:09.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1780741120 bytes 2026-02-20T22:41:09.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1825130496 bytes 2026-02-20T22:41:09.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1881414656 bytes 2026-02-20T22:41:09.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1989692416 bytes 2026-02-20T22:41:09.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2035122176 bytes 2026-02-20T22:41:09.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2058354688 bytes 2026-02-20T22:41:09.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2110912512 bytes 2026-02-20T22:41:09.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2145174528 bytes 2026-02-20T22:41:09.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2196030464 bytes 2026-02-20T22:41:09.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2263006208 bytes 2026-02-20T22:41:09.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2311629824 bytes 2026-02-20T22:41:09.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2437741568 bytes 2026-02-20T22:41:09.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2462087168 bytes 2026-02-20T22:41:09.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2484209664 bytes 2026-02-20T22:41:09.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2546489344 bytes 2026-02-20T22:41:09.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2547105792 bytes 2026-02-20T22:41:10.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2582202368 bytes 2026-02-20T22:41:10.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2612993024 bytes 2026-02-20T22:41:10.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2635204608 bytes 2026-02-20T22:41:10.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2704691200 bytes 2026-02-20T22:41:10.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2724093952 bytes 2026-02-20T22:41:10.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2802875392 bytes 2026-02-20T22:41:10.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2867551232 bytes 2026-02-20T22:41:10.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2887913472 bytes 2026-02-20T22:41:10.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2929254400 bytes 2026-02-20T22:41:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2982325248 bytes 2026-02-20T22:41:10.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3075278848 bytes 2026-02-20T22:41:10.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3119287296 bytes 2026-02-20T22:41:10.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3162433536 bytes 2026-02-20T22:41:10.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3202249728 bytes 2026-02-20T22:41:10.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3224908800 bytes 2026-02-20T22:41:10.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3274167296 bytes 2026-02-20T22:41:11.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3291520000 bytes 2026-02-20T22:41:11.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3371971584 bytes 2026-02-20T22:41:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3419909120 bytes 2026-02-20T22:41:11.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3489568768 bytes 2026-02-20T22:41:11.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3519209472 bytes 2026-02-20T22:41:11.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3545361408 bytes 2026-02-20T22:41:11.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3597404160 bytes 2026-02-20T22:41:11.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3610977280 bytes 2026-02-20T22:41:11.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3653074944 bytes 2026-02-20T22:41:11.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3668207616 bytes 2026-02-20T22:41:11.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3719835648 bytes 2026-02-20T22:41:11.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3756351488 bytes 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:41:11.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_33 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_33 128M 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:11.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2026-02-20T22:41:11.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:11.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2026-02-20T22:41:12.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:12.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_33': 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13d97b669c63 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13d97b669c63 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:12.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:11 2026 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:11 2026 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:11 2026 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 127d9d72-c25f-4d41-a45b-12cfcc97b19d 2026-02-20T22:41:12.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:12.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:12.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_33 2026-02-20T22:41:12.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:12.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:12.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:41:12.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:12.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 45361152 bytes 2026-02-20T22:41:12.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 45971456 bytes 2026-02-20T22:41:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 74870784 bytes 2026-02-20T22:41:12.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 79192064 bytes 2026-02-20T22:41:12.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 83050496 bytes 2026-02-20T22:41:12.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 100859904 bytes 2026-02-20T22:41:13.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 137194496 bytes 2026-02-20T22:41:13.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 170738688 bytes 2026-02-20T22:41:13.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 171352064 bytes 2026-02-20T22:41:13.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 192616448 bytes 2026-02-20T22:41:13.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 232551424 bytes 2026-02-20T22:41:13.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 251725824 bytes 2026-02-20T22:41:13.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 298707968 bytes 2026-02-20T22:41:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 312353792 bytes 2026-02-20T22:41:13.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 389254144 bytes 2026-02-20T22:41:13.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 415468544 bytes 2026-02-20T22:41:13.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 503101440 bytes 2026-02-20T22:41:13.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 540655616 bytes 2026-02-20T22:41:13.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 541532160 bytes 2026-02-20T22:41:13.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 614356992 bytes 2026-02-20T22:41:13.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 647083008 bytes 2026-02-20T22:41:13.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 686618624 bytes 2026-02-20T22:41:13.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 718166016 bytes 2026-02-20T22:41:13.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 741011456 bytes 2026-02-20T22:41:14.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 807052288 bytes 2026-02-20T22:41:14.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 807646208 bytes 2026-02-20T22:41:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 903613440 bytes 2026-02-20T22:41:14.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 963345408 bytes 2026-02-20T22:41:14.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 995946496 bytes 2026-02-20T22:41:14.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1004360704 bytes 2026-02-20T22:41:14.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1063259136 bytes 2026-02-20T22:41:14.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1151949824 bytes 2026-02-20T22:41:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1164579840 bytes 2026-02-20T22:41:14.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1165194240 bytes 2026-02-20T22:41:14.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1178359808 bytes 2026-02-20T22:41:14.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1265086464 bytes 2026-02-20T22:41:14.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1301880832 bytes 2026-02-20T22:41:14.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1362136064 bytes 2026-02-20T22:41:15.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1425303552 bytes 2026-02-20T22:41:15.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1472645120 bytes 2026-02-20T22:41:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1600475136 bytes 2026-02-20T22:41:15.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1702256640 bytes 2026-02-20T22:41:15.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1809575936 bytes 2026-02-20T22:41:15.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1850610688 bytes 2026-02-20T22:41:15.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1919033344 bytes 2026-02-20T22:41:15.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1975104512 bytes 2026-02-20T22:41:15.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1976224768 bytes 2026-02-20T22:41:15.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1996150784 bytes 2026-02-20T22:41:15.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2014596096 bytes 2026-02-20T22:41:15.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2093201408 bytes 2026-02-20T22:41:15.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2118901760 bytes 2026-02-20T22:41:15.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2188990464 bytes 2026-02-20T22:41:15.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2246088704 bytes 2026-02-20T22:41:16.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2269586432 bytes 2026-02-20T22:41:16.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2294887424 bytes 2026-02-20T22:41:16.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2339619840 bytes 2026-02-20T22:41:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2405211136 bytes 2026-02-20T22:41:16.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2425152512 bytes 2026-02-20T22:41:16.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2475811840 bytes 2026-02-20T22:41:16.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2562741248 bytes 2026-02-20T22:41:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2593221632 bytes 2026-02-20T22:41:16.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2594922496 bytes 2026-02-20T22:41:16.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2672211968 bytes 2026-02-20T22:41:16.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2692502528 bytes 2026-02-20T22:41:16.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2703959040 bytes 2026-02-20T22:41:16.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2733619200 bytes 2026-02-20T22:41:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2783937536 bytes 2026-02-20T22:41:16.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2791652352 bytes 2026-02-20T22:41:16.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2859959296 bytes 2026-02-20T22:41:16.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2868997120 bytes 2026-02-20T22:41:16.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2887222272 bytes 2026-02-20T22:41:16.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2921350144 bytes 2026-02-20T22:41:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2986136576 bytes 2026-02-20T22:41:17.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3058670592 bytes 2026-02-20T22:41:17.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3061220352 bytes 2026-02-20T22:41:17.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3061841920 bytes 2026-02-20T22:41:17.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3150236672 bytes 2026-02-20T22:41:17.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3170976768 bytes 2026-02-20T22:41:17.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3192994816 bytes 2026-02-20T22:41:17.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3227357184 bytes 2026-02-20T22:41:17.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3294068736 bytes 2026-02-20T22:41:17.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3349781504 bytes 2026-02-20T22:41:17.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3393244160 bytes 2026-02-20T22:41:17.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3433004032 bytes 2026-02-20T22:41:17.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3525253120 bytes 2026-02-20T22:41:17.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3539613696 bytes 2026-02-20T22:41:17.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3600150528 bytes 2026-02-20T22:41:17.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3658319872 bytes 2026-02-20T22:41:17.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3661386752 bytes 2026-02-20T22:41:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3683876864 bytes 2026-02-20T22:41:18.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3793082368 bytes 2026-02-20T22:41:18.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3817531392 bytes 2026-02-20T22:41:18.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3841379328 bytes 2026-02-20T22:41:18.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3843571712 bytes 2026-02-20T22:41:18.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3910145024 bytes 2026-02-20T22:41:18.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3932514304 bytes 2026-02-20T22:41:18.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3964300288 bytes 2026-02-20T22:41:18.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3986023424 bytes 2026-02-20T22:41:18.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4013147136 bytes 2026-02-20T22:41:18.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4030410752 bytes 2026-02-20T22:41:18.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4045169664 bytes 2026-02-20T22:41:18.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4088670208 bytes 2026-02-20T22:41:18.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4120996864 bytes 2026-02-20T22:41:18.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4140552192 bytes 2026-02-20T22:41:18.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4213443584 bytes 2026-02-20T22:41:18.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4240806912 bytes 2026-02-20T22:41:18.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4257306624 bytes 2026-02-20T22:41:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4266969088 bytes 2026-02-20T22:41:19.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4281289728 bytes 2026-02-20T22:41:19.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4337273856 bytes 2026-02-20T22:41:19.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4439223296 bytes 2026-02-20T22:41:19.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4443089920 bytes 2026-02-20T22:41:19.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4479491072 bytes 2026-02-20T22:41:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4488352768 bytes 2026-02-20T22:41:19.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4525853696 bytes 2026-02-20T22:41:19.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4579983360 bytes 2026-02-20T22:41:19.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4605497344 bytes 2026-02-20T22:41:19.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4609324032 bytes 2026-02-20T22:41:19.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4659604480 bytes 2026-02-20T22:41:19.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4682687488 bytes 2026-02-20T22:41:19.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4791002112 bytes 2026-02-20T22:41:19.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4888514560 bytes 2026-02-20T22:41:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4938534912 bytes 2026-02-20T22:41:19.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4975674368 bytes 2026-02-20T22:41:20.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5016894464 bytes 2026-02-20T22:41:20.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5109634048 bytes 2026-02-20T22:41:20.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5113113600 bytes 2026-02-20T22:41:20.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5136633856 bytes 2026-02-20T22:41:20.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5168739328 bytes 2026-02-20T22:41:20.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5238601728 bytes 2026-02-20T22:41:20.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5260009472 bytes 2026-02-20T22:41:20.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5260621824 bytes 2026-02-20T22:41:20.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5290155008 bytes 2026-02-20T22:41:20.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5334333440 bytes 2026-02-20T22:41:20.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5375974400 bytes 2026-02-20T22:41:20.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5415508992 bytes 2026-02-20T22:41:20.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5452089344 bytes 2026-02-20T22:41:20.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5539596288 bytes 2026-02-20T22:41:20.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5592427520 bytes 2026-02-20T22:41:20.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5603683328 bytes 2026-02-20T22:41:20.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5613353984 bytes 2026-02-20T22:41:21.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5689401344 bytes 2026-02-20T22:41:21.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5796038656 bytes 2026-02-20T22:41:21.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5851249664 bytes 2026-02-20T22:41:21.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5853414400 bytes 2026-02-20T22:41:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5875863552 bytes 2026-02-20T22:41:21.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5876492288 bytes 2026-02-20T22:41:21.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5900348416 bytes 2026-02-20T22:41:21.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 5986257920 bytes 2026-02-20T22:41:21.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 5991158784 bytes 2026-02-20T22:41:21.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6037946368 bytes 2026-02-20T22:41:21.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6086780928 bytes 2026-02-20T22:41:21.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6175121408 bytes 2026-02-20T22:41:21.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6231035904 bytes 2026-02-20T22:41:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6237438976 bytes 2026-02-20T22:41:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6284247040 bytes 2026-02-20T22:41:22.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6340673536 bytes 2026-02-20T22:41:22.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6380699648 bytes 2026-02-20T22:41:22.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6482288640 bytes 2026-02-20T22:41:22.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6497707008 bytes 2026-02-20T22:41:22.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6547252224 bytes 2026-02-20T22:41:22.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6620957696 bytes 2026-02-20T22:41:22.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6638651392 bytes 2026-02-20T22:41:22.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6698262528 bytes 2026-02-20T22:41:22.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6757202944 bytes 2026-02-20T22:41:22.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6770239488 bytes 2026-02-20T22:41:22.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6783657984 bytes 2026-02-20T22:41:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:41:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:41:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:41:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:22.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_34 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_34 128M 2026-02-20T22:41:22.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:22.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2026-02-20T22:41:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:22.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2026-02-20T22:41:22.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:22.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_34': 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13e1a08b6883 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13e1a08b6883 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:22.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:22 2026 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:22 2026 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:22 2026 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 4f963951-2e0e-4d72-bfa8-798685ade5d5 2026-02-20T22:41:22.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:22.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:22.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_34 2026-02-20T22:41:22.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:22.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:22.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:41:22.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:22.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:41:22.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:22.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:23.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 42872832 bytes 2026-02-20T22:41:23.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 43500544 bytes 2026-02-20T22:41:23.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 67502080 bytes 2026-02-20T22:41:23.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 79289344 bytes 2026-02-20T22:41:23.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 151038976 bytes 2026-02-20T22:41:23.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 181147648 bytes 2026-02-20T22:41:23.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 264899584 bytes 2026-02-20T22:41:23.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 323760128 bytes 2026-02-20T22:41:23.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 340155392 bytes 2026-02-20T22:41:23.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 372288512 bytes 2026-02-20T22:41:23.539 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7751761792518862) > weight (0.25) 2026-02-20T22:41:23.539 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-20T22:41:23.539 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-20T22:41:23.539 INFO:tasks.rbd_mirror_thrash:waiting for 2.047666187076386 secs before reviving daemons 2026-02-20T22:41:23.540 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:41:23.535+0000 7f190ddd7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 23602) UID: 1000 2026-02-20T22:41:23.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 419393536 bytes 2026-02-20T22:41:23.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 469123072 bytes 2026-02-20T22:41:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 493557760 bytes 2026-02-20T22:41:23.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 541690880 bytes 2026-02-20T22:41:23.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 552075264 bytes 2026-02-20T22:41:23.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 585749504 bytes 2026-02-20T22:41:23.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 642670592 bytes 2026-02-20T22:41:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 690401280 bytes 2026-02-20T22:41:24.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 746128384 bytes 2026-02-20T22:41:24.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 777692160 bytes 2026-02-20T22:41:24.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 791610368 bytes 2026-02-20T22:41:24.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 859648000 bytes 2026-02-20T22:41:24.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 882825216 bytes 2026-02-20T22:41:24.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 900834304 bytes 2026-02-20T22:41:24.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 911782912 bytes 2026-02-20T22:41:24.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 966705152 bytes 2026-02-20T22:41:24.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 971638784 bytes 2026-02-20T22:41:24.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1016074240 bytes 2026-02-20T22:41:24.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1065586688 bytes 2026-02-20T22:41:24.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1121560576 bytes 2026-02-20T22:41:24.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1174818816 bytes 2026-02-20T22:41:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1199885312 bytes 2026-02-20T22:41:24.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1246733312 bytes 2026-02-20T22:41:24.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1272602624 bytes 2026-02-20T22:41:24.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1326176256 bytes 2026-02-20T22:41:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1409420288 bytes 2026-02-20T22:41:25.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1499786240 bytes 2026-02-20T22:41:25.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1582593024 bytes 2026-02-20T22:41:25.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1707229184 bytes 2026-02-20T22:41:25.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1757315072 bytes 2026-02-20T22:41:25.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1783247872 bytes 2026-02-20T22:41:25.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1809476608 bytes 2026-02-20T22:41:25.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1834602496 bytes 2026-02-20T22:41:25.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1840320512 bytes 2026-02-20T22:41:25.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1896212480 bytes 2026-02-20T22:41:25.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1896865792 bytes 2026-02-20T22:41:25.588 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-20T22:41:25.588 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:41:25.589 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-20T22:41:25.589 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-20T22:41:25.590 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-20T22:41:25.592 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-20T22:41:25.592 INFO:tasks.rbd_mirror_thrash:waiting for 119.47953145808356 secs before thrashing 2026-02-20T22:41:25.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1936306176 bytes 2026-02-20T22:41:25.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1998623744 bytes 2026-02-20T22:41:25.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2051246080 bytes 2026-02-20T22:41:25.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2051894272 bytes 2026-02-20T22:41:25.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2115994624 bytes 2026-02-20T22:41:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2180093952 bytes 2026-02-20T22:41:26.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2238693376 bytes 2026-02-20T22:41:26.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2245739520 bytes 2026-02-20T22:41:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2326009856 bytes 2026-02-20T22:41:26.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2328623104 bytes 2026-02-20T22:41:26.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2333361152 bytes 2026-02-20T22:41:26.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2393845760 bytes 2026-02-20T22:41:26.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2482990080 bytes 2026-02-20T22:41:26.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2591848448 bytes 2026-02-20T22:41:26.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2596045824 bytes 2026-02-20T22:41:26.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2662289408 bytes 2026-02-20T22:41:26.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2723503104 bytes 2026-02-20T22:41:26.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2792504320 bytes 2026-02-20T22:41:26.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2832521216 bytes 2026-02-20T22:41:26.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2918144000 bytes 2026-02-20T22:41:26.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2979816448 bytes 2026-02-20T22:41:26.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2980434944 bytes 2026-02-20T22:41:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3051237376 bytes 2026-02-20T22:41:27.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3098013696 bytes 2026-02-20T22:41:27.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3121138688 bytes 2026-02-20T22:41:27.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3178107904 bytes 2026-02-20T22:41:27.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3209859072 bytes 2026-02-20T22:41:27.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3289663488 bytes 2026-02-20T22:41:27.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3360839680 bytes 2026-02-20T22:41:27.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3391071232 bytes 2026-02-20T22:41:27.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3468143616 bytes 2026-02-20T22:41:27.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3492914176 bytes 2026-02-20T22:41:27.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3496617984 bytes 2026-02-20T22:41:27.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3519987712 bytes 2026-02-20T22:41:27.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3577863168 bytes 2026-02-20T22:41:27.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3594637312 bytes 2026-02-20T22:41:27.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3623546880 bytes 2026-02-20T22:41:27.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3696794624 bytes 2026-02-20T22:41:27.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3721848832 bytes 2026-02-20T22:41:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3733047296 bytes 2026-02-20T22:41:27.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3747268608 bytes 2026-02-20T22:41:28.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3780528128 bytes 2026-02-20T22:41:28.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3862008832 bytes 2026-02-20T22:41:28.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3921405952 bytes 2026-02-20T22:41:28.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3996616704 bytes 2026-02-20T22:41:28.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4029895680 bytes 2026-02-20T22:41:28.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4049341440 bytes 2026-02-20T22:41:28.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4097423360 bytes 2026-02-20T22:41:28.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4158394368 bytes 2026-02-20T22:41:28.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4176886784 bytes 2026-02-20T22:41:28.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4198348800 bytes 2026-02-20T22:41:28.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4257397760 bytes 2026-02-20T22:41:28.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4332229632 bytes 2026-02-20T22:41:28.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4379049984 bytes 2026-02-20T22:41:28.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4471915520 bytes 2026-02-20T22:41:28.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4529995776 bytes 2026-02-20T22:41:29.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4559336448 bytes 2026-02-20T22:41:29.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4668435456 bytes 2026-02-20T22:41:29.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4701106176 bytes 2026-02-20T22:41:29.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4739294208 bytes 2026-02-20T22:41:29.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4790137856 bytes 2026-02-20T22:41:29.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4898367488 bytes 2026-02-20T22:41:29.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4981500928 bytes 2026-02-20T22:41:29.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5020073984 bytes 2026-02-20T22:41:29.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5085585408 bytes 2026-02-20T22:41:29.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5124403200 bytes 2026-02-20T22:41:29.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5170953216 bytes 2026-02-20T22:41:29.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5228104704 bytes 2026-02-20T22:41:29.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5266095104 bytes 2026-02-20T22:41:29.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5300170752 bytes 2026-02-20T22:41:29.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5333298176 bytes 2026-02-20T22:41:30.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5403513856 bytes 2026-02-20T22:41:30.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5443415040 bytes 2026-02-20T22:41:30.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5467498496 bytes 2026-02-20T22:41:30.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5562049536 bytes 2026-02-20T22:41:30.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5638147072 bytes 2026-02-20T22:41:30.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5654594560 bytes 2026-02-20T22:41:30.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5743321088 bytes 2026-02-20T22:41:30.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5829306368 bytes 2026-02-20T22:41:30.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5852477440 bytes 2026-02-20T22:41:30.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5939311616 bytes 2026-02-20T22:41:30.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5972916224 bytes 2026-02-20T22:41:30.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 6052203520 bytes 2026-02-20T22:41:30.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6157247488 bytes 2026-02-20T22:41:30.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6221122560 bytes 2026-02-20T22:41:30.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6237229056 bytes 2026-02-20T22:41:30.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6268861440 bytes 2026-02-20T22:41:31.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6291998720 bytes 2026-02-20T22:41:31.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6322829312 bytes 2026-02-20T22:41:31.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6360312832 bytes 2026-02-20T22:41:31.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6387518464 bytes 2026-02-20T22:41:31.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6430582784 bytes 2026-02-20T22:41:31.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6508192768 bytes 2026-02-20T22:41:31.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6516178944 bytes 2026-02-20T22:41:31.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6552341504 bytes 2026-02-20T22:41:31.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6563987456 bytes 2026-02-20T22:41:31.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6583834624 bytes 2026-02-20T22:41:31.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6623753216 bytes 2026-02-20T22:41:31.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6635266048 bytes 2026-02-20T22:41:31.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6679954432 bytes 2026-02-20T22:41:31.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6718564352 bytes 2026-02-20T22:41:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6765547520 bytes 2026-02-20T22:41:31.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6840936448 bytes 2026-02-20T22:41:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6887177216 bytes 2026-02-20T22:41:31.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6931110912 bytes 2026-02-20T22:41:32.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6965213184 bytes 2026-02-20T22:41:32.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 7054792704 bytes 2026-02-20T22:41:32.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 7134406656 bytes 2026-02-20T22:41:32.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7167293440 bytes 2026-02-20T22:41:32.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7204750336 bytes 2026-02-20T22:41:32.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7322571776 bytes 2026-02-20T22:41:32.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7390710784 bytes 2026-02-20T22:41:32.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7455985664 bytes 2026-02-20T22:41:32.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7501416448 bytes 2026-02-20T22:41:32.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7554725888 bytes 2026-02-20T22:41:32.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7585734656 bytes 2026-02-20T22:41:32.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7587847168 bytes 2026-02-20T22:41:32.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7645872128 bytes 2026-02-20T22:41:32.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7670315008 bytes 2026-02-20T22:41:32.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7720170496 bytes 2026-02-20T22:41:32.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7739031552 bytes 2026-02-20T22:41:33.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7808430080 bytes 2026-02-20T22:41:33.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7876613120 bytes 2026-02-20T22:41:33.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7913011200 bytes 2026-02-20T22:41:33.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7918129152 bytes 2026-02-20T22:41:33.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7957947392 bytes 2026-02-20T22:41:33.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7977974784 bytes 2026-02-20T22:41:33.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 8009814016 bytes 2026-02-20T22:41:33.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 8082616320 bytes 2026-02-20T22:41:33.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 8103544832 bytes 2026-02-20T22:41:33.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 8168135680 bytes 2026-02-20T22:41:33.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 8212546560 bytes 2026-02-20T22:41:33.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 8292893696 bytes 2026-02-20T22:41:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 8328198144 bytes 2026-02-20T22:41:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8389074944 bytes 2026-02-20T22:41:33.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8467524608 bytes 2026-02-20T22:41:33.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8495541248 bytes 2026-02-20T22:41:33.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8543682560 bytes 2026-02-20T22:41:34.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8603063296 bytes 2026-02-20T22:41:34.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8644400128 bytes 2026-02-20T22:41:34.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8684999680 bytes 2026-02-20T22:41:34.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8755530752 bytes 2026-02-20T22:41:34.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8827152384 bytes 2026-02-20T22:41:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8879432704 bytes 2026-02-20T22:41:34.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8964104192 bytes 2026-02-20T22:41:34.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8976955392 bytes 2026-02-20T22:41:34.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 9001290752 bytes 2026-02-20T22:41:34.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 9004602368 bytes 2026-02-20T22:41:34.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 9005196288 bytes 2026-02-20T22:41:34.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 9089191936 bytes 2026-02-20T22:41:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 9142011904 bytes 2026-02-20T22:41:34.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 9177940992 bytes 2026-02-20T22:41:34.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 9213789184 bytes 2026-02-20T22:41:34.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 9278891008 bytes 2026-02-20T22:41:35.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 9341523968 bytes 2026-02-20T22:41:35.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 9376371712 bytes 2026-02-20T22:41:35.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 9425121280 bytes 2026-02-20T22:41:35.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9484584960 bytes 2026-02-20T22:41:35.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9549762560 bytes 2026-02-20T22:41:35.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9613805568 bytes 2026-02-20T22:41:35.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9696429056 bytes 2026-02-20T22:41:35.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9728294912 bytes 2026-02-20T22:41:35.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9758204928 bytes 2026-02-20T22:41:35.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9764858880 bytes 2026-02-20T22:41:35.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9815150592 bytes 2026-02-20T22:41:35.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9845368832 bytes 2026-02-20T22:41:35.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9931291648 bytes 2026-02-20T22:41:35.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9974227968 bytes 2026-02-20T22:41:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 10039758848 bytes 2026-02-20T22:41:36.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 10086238208 bytes 2026-02-20T22:41:36.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 10195464192 bytes 2026-02-20T22:41:36.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 10234140672 bytes 2026-02-20T22:41:36.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 10250799104 bytes 2026-02-20T22:41:36.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 10289805312 bytes 2026-02-20T22:41:36.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 10324613120 bytes 2026-02-20T22:41:36.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 10344571904 bytes 2026-02-20T22:41:36.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 10429360128 bytes 2026-02-20T22:41:36.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 10449157120 bytes 2026-02-20T22:41:36.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 10480785408 bytes 2026-02-20T22:41:36.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 10515367936 bytes 2026-02-20T22:41:36.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 10521897984 bytes 2026-02-20T22:41:36.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 10560062464 bytes 2026-02-20T22:41:36.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_35 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_35 128M 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:41:36.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:41:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:41:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:41:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2026-02-20T22:41:37.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:37.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_35': 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13f52be2edcf 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13f52be2edcf 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:36 2026 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:36 2026 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:36 2026 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7ab82fe0-366c-45d8-adf9-a17acd1015ea 2026-02-20T22:41:37.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_35 2026-02-20T22:41:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:41:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:41:37.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:37.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:38.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 74995712 bytes 2026-02-20T22:41:38.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 99603456 bytes 2026-02-20T22:41:38.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 158567424 bytes 2026-02-20T22:41:38.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 185851904 bytes 2026-02-20T22:41:38.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 256875520 bytes 2026-02-20T22:41:38.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 274995200 bytes 2026-02-20T22:41:38.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 372445184 bytes 2026-02-20T22:41:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 379985920 bytes 2026-02-20T22:41:38.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 453178368 bytes 2026-02-20T22:41:38.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 474481664 bytes 2026-02-20T22:41:38.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 504363008 bytes 2026-02-20T22:41:38.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 523770880 bytes 2026-02-20T22:41:38.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 564328448 bytes 2026-02-20T22:41:38.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 613492736 bytes 2026-02-20T22:41:38.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 651127808 bytes 2026-02-20T22:41:38.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 694761472 bytes 2026-02-20T22:41:38.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 700409856 bytes 2026-02-20T22:41:38.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 737609728 bytes 2026-02-20T22:41:39.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 746757120 bytes 2026-02-20T22:41:39.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 768553984 bytes 2026-02-20T22:41:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 791616512 bytes 2026-02-20T22:41:39.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 805076992 bytes 2026-02-20T22:41:39.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 868116480 bytes 2026-02-20T22:41:39.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 877045760 bytes 2026-02-20T22:41:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 911596544 bytes 2026-02-20T22:41:39.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1047682048 bytes 2026-02-20T22:41:39.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1081367552 bytes 2026-02-20T22:41:39.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1095119872 bytes 2026-02-20T22:41:39.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1168802816 bytes 2026-02-20T22:41:39.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1184388096 bytes 2026-02-20T22:41:39.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1286855680 bytes 2026-02-20T22:41:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1327266816 bytes 2026-02-20T22:41:40.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1394403328 bytes 2026-02-20T22:41:40.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1412724736 bytes 2026-02-20T22:41:40.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1428228096 bytes 2026-02-20T22:41:40.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1534484480 bytes 2026-02-20T22:41:40.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1575584768 bytes 2026-02-20T22:41:40.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1632027648 bytes 2026-02-20T22:41:40.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1687542784 bytes 2026-02-20T22:41:40.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1718796288 bytes 2026-02-20T22:41:40.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1735163904 bytes 2026-02-20T22:41:40.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1739620352 bytes 2026-02-20T22:41:40.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1781043200 bytes 2026-02-20T22:41:40.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1787144192 bytes 2026-02-20T22:41:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1833781248 bytes 2026-02-20T22:41:40.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1856786432 bytes 2026-02-20T22:41:40.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1869272064 bytes 2026-02-20T22:41:40.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1940340736 bytes 2026-02-20T22:41:40.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2007187456 bytes 2026-02-20T22:41:41.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2084238336 bytes 2026-02-20T22:41:41.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2122323968 bytes 2026-02-20T22:41:41.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2140667904 bytes 2026-02-20T22:41:41.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2241814528 bytes 2026-02-20T22:41:41.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2332449792 bytes 2026-02-20T22:41:41.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2447588352 bytes 2026-02-20T22:41:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2513352704 bytes 2026-02-20T22:41:41.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2603023360 bytes 2026-02-20T22:41:41.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2641236992 bytes 2026-02-20T22:41:41.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2648266752 bytes 2026-02-20T22:41:41.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2687638528 bytes 2026-02-20T22:41:41.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2751617024 bytes 2026-02-20T22:41:41.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2818874368 bytes 2026-02-20T22:41:41.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2857928704 bytes 2026-02-20T22:41:41.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2903278592 bytes 2026-02-20T22:41:42.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2940711936 bytes 2026-02-20T22:41:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2945693696 bytes 2026-02-20T22:41:42.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3052915712 bytes 2026-02-20T22:41:42.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3077254144 bytes 2026-02-20T22:41:42.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3099623424 bytes 2026-02-20T22:41:42.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3114777600 bytes 2026-02-20T22:41:42.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3149724672 bytes 2026-02-20T22:41:42.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3248876544 bytes 2026-02-20T22:41:42.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3309137920 bytes 2026-02-20T22:41:42.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3325597696 bytes 2026-02-20T22:41:42.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3382278144 bytes 2026-02-20T22:41:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3474990080 bytes 2026-02-20T22:41:42.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3494591488 bytes 2026-02-20T22:41:42.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3500279808 bytes 2026-02-20T22:41:42.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3590483968 bytes 2026-02-20T22:41:42.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3641006080 bytes 2026-02-20T22:41:42.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3643514880 bytes 2026-02-20T22:41:43.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3662158848 bytes 2026-02-20T22:41:43.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3716587520 bytes 2026-02-20T22:41:43.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3788465152 bytes 2026-02-20T22:41:43.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3813820416 bytes 2026-02-20T22:41:43.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3912828928 bytes 2026-02-20T22:41:43.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3976724480 bytes 2026-02-20T22:41:43.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4016939008 bytes 2026-02-20T22:41:43.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4124293120 bytes 2026-02-20T22:41:43.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4208584704 bytes 2026-02-20T22:41:43.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4246786048 bytes 2026-02-20T22:41:43.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4272510976 bytes 2026-02-20T22:41:43.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4304162816 bytes 2026-02-20T22:41:43.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4327955456 bytes 2026-02-20T22:41:44.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4341890048 bytes 2026-02-20T22:41:44.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4378701824 bytes 2026-02-20T22:41:44.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4456243200 bytes 2026-02-20T22:41:44.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4479036416 bytes 2026-02-20T22:41:44.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4524290048 bytes 2026-02-20T22:41:44.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4544509952 bytes 2026-02-20T22:41:44.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4590055424 bytes 2026-02-20T22:41:44.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4590665728 bytes 2026-02-20T22:41:44.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4665632768 bytes 2026-02-20T22:41:44.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4690027520 bytes 2026-02-20T22:41:44.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4716425216 bytes 2026-02-20T22:41:44.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4785983488 bytes 2026-02-20T22:41:44.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4796222464 bytes 2026-02-20T22:41:44.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4838925312 bytes 2026-02-20T22:41:44.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4839528448 bytes 2026-02-20T22:41:44.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4903242752 bytes 2026-02-20T22:41:44.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4941669376 bytes 2026-02-20T22:41:44.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5004812288 bytes 2026-02-20T22:41:45.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5040418816 bytes 2026-02-20T22:41:45.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5146791936 bytes 2026-02-20T22:41:45.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5192501248 bytes 2026-02-20T22:41:45.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5254634496 bytes 2026-02-20T22:41:45.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5375396864 bytes 2026-02-20T22:41:45.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5433463808 bytes 2026-02-20T22:41:45.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5474809856 bytes 2026-02-20T22:41:45.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5518033920 bytes 2026-02-20T22:41:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5547799552 bytes 2026-02-20T22:41:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5582089216 bytes 2026-02-20T22:41:45.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5638367232 bytes 2026-02-20T22:41:45.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5677876224 bytes 2026-02-20T22:41:45.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5744373760 bytes 2026-02-20T22:41:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5777011712 bytes 2026-02-20T22:41:45.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5855086592 bytes 2026-02-20T22:41:46.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5893764096 bytes 2026-02-20T22:41:46.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5915167744 bytes 2026-02-20T22:41:46.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5951295488 bytes 2026-02-20T22:41:46.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5989924864 bytes 2026-02-20T22:41:46.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6025533440 bytes 2026-02-20T22:41:46.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6084791296 bytes 2026-02-20T22:41:46.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6159159296 bytes 2026-02-20T22:41:46.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6227048448 bytes 2026-02-20T22:41:46.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6291628032 bytes 2026-02-20T22:41:46.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6318482432 bytes 2026-02-20T22:41:46.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6356801536 bytes 2026-02-20T22:41:46.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6383005696 bytes 2026-02-20T22:41:46.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6442027008 bytes 2026-02-20T22:41:46.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6481759232 bytes 2026-02-20T22:41:46.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6507190272 bytes 2026-02-20T22:41:47.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6550986752 bytes 2026-02-20T22:41:47.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6618880000 bytes 2026-02-20T22:41:47.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6679899136 bytes 2026-02-20T22:41:47.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6713374720 bytes 2026-02-20T22:41:47.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6760993792 bytes 2026-02-20T22:41:47.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6770557952 bytes 2026-02-20T22:41:47.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6868465664 bytes 2026-02-20T22:41:47.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6924969984 bytes 2026-02-20T22:41:47.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6976875520 bytes 2026-02-20T22:41:47.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 7032630272 bytes 2026-02-20T22:41:47.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 7033234432 bytes 2026-02-20T22:41:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 7088545792 bytes 2026-02-20T22:41:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7166879744 bytes 2026-02-20T22:41:47.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7172513792 bytes 2026-02-20T22:41:47.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7249090560 bytes 2026-02-20T22:41:48.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7325063168 bytes 2026-02-20T22:41:48.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7365604352 bytes 2026-02-20T22:41:48.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7377243136 bytes 2026-02-20T22:41:48.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7403999232 bytes 2026-02-20T22:41:48.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7434380288 bytes 2026-02-20T22:41:48.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7453652992 bytes 2026-02-20T22:41:48.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7500368896 bytes 2026-02-20T22:41:48.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7527159808 bytes 2026-02-20T22:41:48.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7549265920 bytes 2026-02-20T22:41:48.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7591684096 bytes 2026-02-20T22:41:48.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7600245760 bytes 2026-02-20T22:41:48.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7625620480 bytes 2026-02-20T22:41:48.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7650477056 bytes 2026-02-20T22:41:48.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7669145600 bytes 2026-02-20T22:41:48.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7692204032 bytes 2026-02-20T22:41:48.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7692822528 bytes 2026-02-20T22:41:48.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7745406976 bytes 2026-02-20T22:41:48.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7812970496 bytes 2026-02-20T22:41:48.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7875756032 bytes 2026-02-20T22:41:48.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7878881280 bytes 2026-02-20T22:41:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7936120832 bytes 2026-02-20T22:41:49.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7936744448 bytes 2026-02-20T22:41:49.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7979171840 bytes 2026-02-20T22:41:49.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8066606080 bytes 2026-02-20T22:41:49.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8104644608 bytes 2026-02-20T22:41:49.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8162477056 bytes 2026-02-20T22:41:49.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8186848256 bytes 2026-02-20T22:41:49.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8272909312 bytes 2026-02-20T22:41:49.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8304613376 bytes 2026-02-20T22:41:49.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8403748864 bytes 2026-02-20T22:41:49.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8438472704 bytes 2026-02-20T22:41:49.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8521719808 bytes 2026-02-20T22:41:49.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8596312064 bytes 2026-02-20T22:41:49.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8619186176 bytes 2026-02-20T22:41:49.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8672708608 bytes 2026-02-20T22:41:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8692684800 bytes 2026-02-20T22:41:50.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8806051840 bytes 2026-02-20T22:41:50.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8871892992 bytes 2026-02-20T22:41:50.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8955065344 bytes 2026-02-20T22:41:50.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8985338880 bytes 2026-02-20T22:41:50.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 9028529152 bytes 2026-02-20T22:41:50.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 9088199680 bytes 2026-02-20T22:41:50.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 9137513472 bytes 2026-02-20T22:41:50.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 9195444224 bytes 2026-02-20T22:41:50.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 9226210304 bytes 2026-02-20T22:41:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 9242957824 bytes 2026-02-20T22:41:50.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9272473600 bytes 2026-02-20T22:41:50.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9292377088 bytes 2026-02-20T22:41:50.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9322580992 bytes 2026-02-20T22:41:51.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9325027328 bytes 2026-02-20T22:41:51.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9356806144 bytes 2026-02-20T22:41:51.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9373223936 bytes 2026-02-20T22:41:51.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9390900224 bytes 2026-02-20T22:41:51.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9433874432 bytes 2026-02-20T22:41:51.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9442257920 bytes 2026-02-20T22:41:51.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9461164032 bytes 2026-02-20T22:41:51.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9498956800 bytes 2026-02-20T22:41:51.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9552172032 bytes 2026-02-20T22:41:51.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9614672896 bytes 2026-02-20T22:41:51.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9636951040 bytes 2026-02-20T22:41:51.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9712814080 bytes 2026-02-20T22:41:51.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9713427456 bytes 2026-02-20T22:41:51.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9780629504 bytes 2026-02-20T22:41:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9863471104 bytes 2026-02-20T22:41:51.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9928847360 bytes 2026-02-20T22:41:51.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9935472640 bytes 2026-02-20T22:41:51.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9955983360 bytes 2026-02-20T22:41:51.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9983827968 bytes 2026-02-20T22:41:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:41:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_36 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:51.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_36 128M 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:51.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:41:51.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2026-02-20T22:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:41:52.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:41:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2026-02-20T22:41:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:41:52.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2026-02-20T22:41:52.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_36': 2026-02-20T22:41:52.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:41:52.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:41:52.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 13fe10817c56 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.13fe10817c56 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:51 2026 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:51 2026 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:51 2026 2026-02-20T22:41:52.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:41:52.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:41:52.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 685defec-9ba9-4ca2-8daa-bf3c9cad1fd1 2026-02-20T22:41:52.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:41:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:41:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_36 2026-02-20T22:41:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:41:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:41:52.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:41:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:41:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:41:52.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:41:52.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:41:52.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 1284096 bytes 2026-02-20T22:41:52.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 65788928 bytes 2026-02-20T22:41:52.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 97975296 bytes 2026-02-20T22:41:52.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 119501824 bytes 2026-02-20T22:41:52.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 187998208 bytes 2026-02-20T22:41:53.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 229654528 bytes 2026-02-20T22:41:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 252055552 bytes 2026-02-20T22:41:53.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 322337792 bytes 2026-02-20T22:41:53.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 368692224 bytes 2026-02-20T22:41:53.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 504708096 bytes 2026-02-20T22:41:53.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 545276928 bytes 2026-02-20T22:41:53.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 594621440 bytes 2026-02-20T22:41:53.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 651111424 bytes 2026-02-20T22:41:53.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 707681280 bytes 2026-02-20T22:41:53.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 745194496 bytes 2026-02-20T22:41:53.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 783050752 bytes 2026-02-20T22:41:53.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 866407424 bytes 2026-02-20T22:41:53.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 962634752 bytes 2026-02-20T22:41:53.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 1027874816 bytes 2026-02-20T22:41:53.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1090111488 bytes 2026-02-20T22:41:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1148681216 bytes 2026-02-20T22:41:54.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1227055104 bytes 2026-02-20T22:41:54.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1264084992 bytes 2026-02-20T22:41:54.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1356734464 bytes 2026-02-20T22:41:54.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1403276288 bytes 2026-02-20T22:41:54.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1462108160 bytes 2026-02-20T22:41:54.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1550834688 bytes 2026-02-20T22:41:54.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1582779392 bytes 2026-02-20T22:41:54.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1662859264 bytes 2026-02-20T22:41:54.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1669085184 bytes 2026-02-20T22:41:54.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1712101376 bytes 2026-02-20T22:41:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1727813632 bytes 2026-02-20T22:41:54.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1782249472 bytes 2026-02-20T22:41:54.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1852978176 bytes 2026-02-20T22:41:54.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1857937408 bytes 2026-02-20T22:41:54.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1858561024 bytes 2026-02-20T22:41:54.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1864150016 bytes 2026-02-20T22:41:55.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1960280064 bytes 2026-02-20T22:41:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 2011609088 bytes 2026-02-20T22:41:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 2070820864 bytes 2026-02-20T22:41:55.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 2158967808 bytes 2026-02-20T22:41:55.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2210069504 bytes 2026-02-20T22:41:55.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2227713024 bytes 2026-02-20T22:41:55.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2268317696 bytes 2026-02-20T22:41:55.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2287476736 bytes 2026-02-20T22:41:55.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2288082944 bytes 2026-02-20T22:41:55.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2313769984 bytes 2026-02-20T22:41:55.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2336084992 bytes 2026-02-20T22:41:55.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2368148480 bytes 2026-02-20T22:41:55.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2401274880 bytes 2026-02-20T22:41:55.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2457565184 bytes 2026-02-20T22:41:55.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2520449024 bytes 2026-02-20T22:41:55.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2539847680 bytes 2026-02-20T22:41:55.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2583434240 bytes 2026-02-20T22:41:55.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2592234496 bytes 2026-02-20T22:41:55.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2597134336 bytes 2026-02-20T22:41:55.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2669886464 bytes 2026-02-20T22:41:56.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2719384576 bytes 2026-02-20T22:41:56.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2765805568 bytes 2026-02-20T22:41:56.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2830715904 bytes 2026-02-20T22:41:56.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2921337856 bytes 2026-02-20T22:41:56.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2988879872 bytes 2026-02-20T22:41:56.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 3018179584 bytes 2026-02-20T22:41:56.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3084238848 bytes 2026-02-20T22:41:56.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3102527488 bytes 2026-02-20T22:41:56.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3198894080 bytes 2026-02-20T22:41:56.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3226883072 bytes 2026-02-20T22:41:56.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3266709504 bytes 2026-02-20T22:41:56.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3306463232 bytes 2026-02-20T22:41:56.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3383166976 bytes 2026-02-20T22:41:56.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3389469696 bytes 2026-02-20T22:41:56.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3437166592 bytes 2026-02-20T22:41:56.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3465862144 bytes 2026-02-20T22:41:57.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3554190336 bytes 2026-02-20T22:41:57.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3582241792 bytes 2026-02-20T22:41:57.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3599811584 bytes 2026-02-20T22:41:57.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3625357312 bytes 2026-02-20T22:41:57.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3683491840 bytes 2026-02-20T22:41:57.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3719411712 bytes 2026-02-20T22:41:57.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3748350976 bytes 2026-02-20T22:41:57.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3826722816 bytes 2026-02-20T22:41:57.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3924219904 bytes 2026-02-20T22:41:57.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3966032896 bytes 2026-02-20T22:41:57.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3990116352 bytes 2026-02-20T22:41:57.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4048547840 bytes 2026-02-20T22:41:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4091310080 bytes 2026-02-20T22:41:57.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4155891712 bytes 2026-02-20T22:41:57.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4166573056 bytes 2026-02-20T22:41:57.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4232787968 bytes 2026-02-20T22:41:57.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4253364224 bytes 2026-02-20T22:41:57.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4296396800 bytes 2026-02-20T22:41:57.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4297861120 bytes 2026-02-20T22:41:58.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4305010688 bytes 2026-02-20T22:41:58.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4354877440 bytes 2026-02-20T22:41:58.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4368667648 bytes 2026-02-20T22:41:58.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4427313152 bytes 2026-02-20T22:41:58.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4469500928 bytes 2026-02-20T22:41:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4518696960 bytes 2026-02-20T22:41:58.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4588884992 bytes 2026-02-20T22:41:58.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4619238400 bytes 2026-02-20T22:41:58.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4650465280 bytes 2026-02-20T22:41:58.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4693531648 bytes 2026-02-20T22:41:58.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4734618624 bytes 2026-02-20T22:41:58.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4784352256 bytes 2026-02-20T22:41:58.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4837280768 bytes 2026-02-20T22:41:58.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4890654720 bytes 2026-02-20T22:41:59.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4951312384 bytes 2026-02-20T22:41:59.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4962656256 bytes 2026-02-20T22:41:59.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4988319744 bytes 2026-02-20T22:41:59.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5043942400 bytes 2026-02-20T22:41:59.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5072959488 bytes 2026-02-20T22:41:59.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5153549312 bytes 2026-02-20T22:41:59.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5191768064 bytes 2026-02-20T22:41:59.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5231409152 bytes 2026-02-20T22:41:59.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5293343744 bytes 2026-02-20T22:41:59.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5331260416 bytes 2026-02-20T22:41:59.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5340551168 bytes 2026-02-20T22:41:59.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5392045056 bytes 2026-02-20T22:41:59.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5419924480 bytes 2026-02-20T22:41:59.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5472883712 bytes 2026-02-20T22:41:59.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5487703040 bytes 2026-02-20T22:41:59.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5564926976 bytes 2026-02-20T22:41:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5591574528 bytes 2026-02-20T22:41:59.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5624688640 bytes 2026-02-20T22:42:00.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5671547904 bytes 2026-02-20T22:42:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5733630976 bytes 2026-02-20T22:42:00.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5764660224 bytes 2026-02-20T22:42:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5827189760 bytes 2026-02-20T22:42:00.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5863985152 bytes 2026-02-20T22:42:00.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5993227264 bytes 2026-02-20T22:42:00.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6036185088 bytes 2026-02-20T22:42:00.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6084070400 bytes 2026-02-20T22:42:00.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6137772032 bytes 2026-02-20T22:42:00.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6207778816 bytes 2026-02-20T22:42:00.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6243219456 bytes 2026-02-20T22:42:00.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6273825792 bytes 2026-02-20T22:42:00.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6282294272 bytes 2026-02-20T22:42:00.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6309851136 bytes 2026-02-20T22:42:00.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6384954368 bytes 2026-02-20T22:42:00.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6444791808 bytes 2026-02-20T22:42:01.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6510089216 bytes 2026-02-20T22:42:01.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6530859008 bytes 2026-02-20T22:42:01.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6616941568 bytes 2026-02-20T22:42:01.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6640615424 bytes 2026-02-20T22:42:01.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6653167616 bytes 2026-02-20T22:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6672460800 bytes 2026-02-20T22:42:01.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6741458944 bytes 2026-02-20T22:42:01.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6834756608 bytes 2026-02-20T22:42:01.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6906565632 bytes 2026-02-20T22:42:01.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6928502784 bytes 2026-02-20T22:42:01.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6929133568 bytes 2026-02-20T22:42:01.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6990277632 bytes 2026-02-20T22:42:01.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6996670464 bytes 2026-02-20T22:42:01.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 7041949696 bytes 2026-02-20T22:42:01.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7090514944 bytes 2026-02-20T22:42:01.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7118952448 bytes 2026-02-20T22:42:01.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7154001920 bytes 2026-02-20T22:42:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7204463616 bytes 2026-02-20T22:42:02.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7221133312 bytes 2026-02-20T22:42:02.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7227122688 bytes 2026-02-20T22:42:02.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7264860160 bytes 2026-02-20T22:42:02.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7285251072 bytes 2026-02-20T22:42:02.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7299782656 bytes 2026-02-20T22:42:02.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7369891840 bytes 2026-02-20T22:42:02.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7391971328 bytes 2026-02-20T22:42:02.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7425602560 bytes 2026-02-20T22:42:02.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7457496064 bytes 2026-02-20T22:42:02.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7557603328 bytes 2026-02-20T22:42:02.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7558227968 bytes 2026-02-20T22:42:02.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7584242688 bytes 2026-02-20T22:42:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7618054144 bytes 2026-02-20T22:42:02.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7665376256 bytes 2026-02-20T22:42:02.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7725841408 bytes 2026-02-20T22:42:02.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7789268992 bytes 2026-02-20T22:42:02.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7828412416 bytes 2026-02-20T22:42:02.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7873552384 bytes 2026-02-20T22:42:03.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7905910784 bytes 2026-02-20T22:42:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7909509120 bytes 2026-02-20T22:42:03.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7922952192 bytes 2026-02-20T22:42:03.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7992193024 bytes 2026-02-20T22:42:03.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8025528320 bytes 2026-02-20T22:42:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8059072512 bytes 2026-02-20T22:42:03.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8101900288 bytes 2026-02-20T22:42:03.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8105534464 bytes 2026-02-20T22:42:03.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8184849408 bytes 2026-02-20T22:42:03.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8193216512 bytes 2026-02-20T22:42:03.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8262377472 bytes 2026-02-20T22:42:03.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8287612928 bytes 2026-02-20T22:42:03.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8355846144 bytes 2026-02-20T22:42:03.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8398630912 bytes 2026-02-20T22:42:03.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8449573888 bytes 2026-02-20T22:42:03.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8496969728 bytes 2026-02-20T22:42:03.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8503734272 bytes 2026-02-20T22:42:04.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8633223168 bytes 2026-02-20T22:42:04.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8709635072 bytes 2026-02-20T22:42:04.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8747053056 bytes 2026-02-20T22:42:04.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8784917504 bytes 2026-02-20T22:42:04.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8866608128 bytes 2026-02-20T22:42:04.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8972235776 bytes 2026-02-20T22:42:04.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8996681728 bytes 2026-02-20T22:42:04.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 9069093888 bytes 2026-02-20T22:42:04.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 9075159040 bytes 2026-02-20T22:42:04.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 9086015488 bytes 2026-02-20T22:42:04.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9172191232 bytes 2026-02-20T22:42:04.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9231115264 bytes 2026-02-20T22:42:04.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9278665728 bytes 2026-02-20T22:42:04.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9324756992 bytes 2026-02-20T22:42:05.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9428934656 bytes 2026-02-20T22:42:05.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9460451328 bytes 2026-02-20T22:42:05.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9477358592 bytes 2026-02-20T22:42:05.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9493390336 bytes 2026-02-20T22:42:05.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9515328512 bytes 2026-02-20T22:42:05.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9588475904 bytes 2026-02-20T22:42:05.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9671372800 bytes 2026-02-20T22:42:05.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9696017408 bytes 2026-02-20T22:42:05.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9758606336 bytes 2026-02-20T22:42:05.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9785802752 bytes 2026-02-20T22:42:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9801505792 bytes 2026-02-20T22:42:05.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9813092352 bytes 2026-02-20T22:42:05.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9849428992 bytes 2026-02-20T22:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_37 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_37 128M 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:05.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:05.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2026-02-20T22:42:05.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:05.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:05.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:05.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2026-02-20T22:42:06.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:06.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2026-02-20T22:42:06.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_37': 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1408fa6c0b72 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1408fa6c0b72 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:06.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:05 2026 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:05 2026 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:05 2026 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1ca65d24-3943-4fec-a288-c4a0270f5328 2026-02-20T22:42:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_37 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:42:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:42:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:06.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:06.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 36205568 bytes 2026-02-20T22:42:07.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 75965440 bytes 2026-02-20T22:42:07.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 152491008 bytes 2026-02-20T22:42:07.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 208750592 bytes 2026-02-20T22:42:07.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 252554240 bytes 2026-02-20T22:42:07.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 277160960 bytes 2026-02-20T22:42:07.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 282566656 bytes 2026-02-20T22:42:07.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 329350144 bytes 2026-02-20T22:42:07.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 387474432 bytes 2026-02-20T22:42:07.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 398885888 bytes 2026-02-20T22:42:07.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 424455168 bytes 2026-02-20T22:42:07.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 458908672 bytes 2026-02-20T22:42:07.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 482809856 bytes 2026-02-20T22:42:07.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 500818944 bytes 2026-02-20T22:42:07.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 568453120 bytes 2026-02-20T22:42:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 589347840 bytes 2026-02-20T22:42:07.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 660511744 bytes 2026-02-20T22:42:07.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 705138688 bytes 2026-02-20T22:42:07.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 724052992 bytes 2026-02-20T22:42:08.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 783865856 bytes 2026-02-20T22:42:08.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 818944000 bytes 2026-02-20T22:42:08.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 866742272 bytes 2026-02-20T22:42:08.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 912311296 bytes 2026-02-20T22:42:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 933851136 bytes 2026-02-20T22:42:08.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 950305792 bytes 2026-02-20T22:42:08.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1003401216 bytes 2026-02-20T22:42:08.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1035254784 bytes 2026-02-20T22:42:08.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1093584896 bytes 2026-02-20T22:42:08.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1166836736 bytes 2026-02-20T22:42:08.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1197202432 bytes 2026-02-20T22:42:08.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1250774016 bytes 2026-02-20T22:42:08.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1319071744 bytes 2026-02-20T22:42:08.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1354582016 bytes 2026-02-20T22:42:08.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1371290624 bytes 2026-02-20T22:42:08.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1427081216 bytes 2026-02-20T22:42:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1442226176 bytes 2026-02-20T22:42:08.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1463347200 bytes 2026-02-20T22:42:09.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1509614592 bytes 2026-02-20T22:42:09.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1567039488 bytes 2026-02-20T22:42:09.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1575831552 bytes 2026-02-20T22:42:09.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1661149184 bytes 2026-02-20T22:42:09.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1682056192 bytes 2026-02-20T22:42:09.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1706382336 bytes 2026-02-20T22:42:09.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1729094656 bytes 2026-02-20T22:42:09.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1746235392 bytes 2026-02-20T22:42:09.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1772829696 bytes 2026-02-20T22:42:09.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1814260736 bytes 2026-02-20T22:42:09.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1832942592 bytes 2026-02-20T22:42:09.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1856169984 bytes 2026-02-20T22:42:09.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1932642304 bytes 2026-02-20T22:42:09.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1942426624 bytes 2026-02-20T22:42:09.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1967437824 bytes 2026-02-20T22:42:09.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2015939584 bytes 2026-02-20T22:42:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2023239680 bytes 2026-02-20T22:42:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2089442304 bytes 2026-02-20T22:42:09.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2131870720 bytes 2026-02-20T22:42:10.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2177191936 bytes 2026-02-20T22:42:10.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2204489728 bytes 2026-02-20T22:42:10.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2256891904 bytes 2026-02-20T22:42:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2320915456 bytes 2026-02-20T22:42:10.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2335152128 bytes 2026-02-20T22:42:10.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2384714752 bytes 2026-02-20T22:42:10.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2431462400 bytes 2026-02-20T22:42:10.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2495398912 bytes 2026-02-20T22:42:10.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2513173504 bytes 2026-02-20T22:42:10.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2579550208 bytes 2026-02-20T22:42:10.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2637700096 bytes 2026-02-20T22:42:10.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2721611776 bytes 2026-02-20T22:42:10.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2740736000 bytes 2026-02-20T22:42:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2789768192 bytes 2026-02-20T22:42:10.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2808382464 bytes 2026-02-20T22:42:10.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2839049216 bytes 2026-02-20T22:42:10.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2875535360 bytes 2026-02-20T22:42:11.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2886899712 bytes 2026-02-20T22:42:11.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2909256704 bytes 2026-02-20T22:42:11.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3002270720 bytes 2026-02-20T22:42:11.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3008773120 bytes 2026-02-20T22:42:11.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3075802112 bytes 2026-02-20T22:42:11.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3117119488 bytes 2026-02-20T22:42:11.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3169617920 bytes 2026-02-20T22:42:11.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3197217792 bytes 2026-02-20T22:42:11.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3249901568 bytes 2026-02-20T22:42:11.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3258008576 bytes 2026-02-20T22:42:11.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3291543552 bytes 2026-02-20T22:42:11.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3327814656 bytes 2026-02-20T22:42:11.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3338572800 bytes 2026-02-20T22:42:11.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3443885056 bytes 2026-02-20T22:42:11.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3466523648 bytes 2026-02-20T22:42:11.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3529792512 bytes 2026-02-20T22:42:11.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3600499712 bytes 2026-02-20T22:42:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3650684928 bytes 2026-02-20T22:42:12.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3734320128 bytes 2026-02-20T22:42:12.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3806846976 bytes 2026-02-20T22:42:12.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3861617664 bytes 2026-02-20T22:42:12.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3911189504 bytes 2026-02-20T22:42:12.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3963625472 bytes 2026-02-20T22:42:12.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3989658624 bytes 2026-02-20T22:42:12.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4078886912 bytes 2026-02-20T22:42:12.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4135313408 bytes 2026-02-20T22:42:12.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4174889984 bytes 2026-02-20T22:42:12.666 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4248118272 bytes 2026-02-20T22:42:12.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4306290688 bytes 2026-02-20T22:42:12.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4332266496 bytes 2026-02-20T22:42:12.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4343705600 bytes 2026-02-20T22:42:12.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4365632512 bytes 2026-02-20T22:42:12.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4373567488 bytes 2026-02-20T22:42:12.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4436611072 bytes 2026-02-20T22:42:13.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4478217216 bytes 2026-02-20T22:42:13.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4532188160 bytes 2026-02-20T22:42:13.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4548903936 bytes 2026-02-20T22:42:13.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4673990656 bytes 2026-02-20T22:42:13.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3213409647795301) > weight (0.25) 2026-02-20T22:42:13.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3634245935471888) > weight (0.25) 2026-02-20T22:42:13.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8952957876111972) > weight (0.25) 2026-02-20T22:42:13.274 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.8044133687230722) > weight (0.25) 2026-02-20T22:42:13.275 INFO:tasks.rbd_mirror_thrash:waiting for 106.79900139500526 secs before thrashing 2026-02-20T22:42:13.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4714904576 bytes 2026-02-20T22:42:13.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4788915200 bytes 2026-02-20T22:42:13.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4812522496 bytes 2026-02-20T22:42:13.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4835160064 bytes 2026-02-20T22:42:13.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4888900608 bytes 2026-02-20T22:42:13.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4934839296 bytes 2026-02-20T22:42:13.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4994662400 bytes 2026-02-20T22:42:13.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5054104576 bytes 2026-02-20T22:42:13.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5054713856 bytes 2026-02-20T22:42:13.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5100654592 bytes 2026-02-20T22:42:13.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5124465664 bytes 2026-02-20T22:42:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5184514048 bytes 2026-02-20T22:42:13.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5201959936 bytes 2026-02-20T22:42:14.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5256974336 bytes 2026-02-20T22:42:14.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5284657152 bytes 2026-02-20T22:42:14.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5304866816 bytes 2026-02-20T22:42:14.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5313837056 bytes 2026-02-20T22:42:14.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5336726528 bytes 2026-02-20T22:42:14.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5405789184 bytes 2026-02-20T22:42:14.359 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5472131072 bytes 2026-02-20T22:42:14.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5511953408 bytes 2026-02-20T22:42:14.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5552798720 bytes 2026-02-20T22:42:14.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5595021312 bytes 2026-02-20T22:42:14.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5614244864 bytes 2026-02-20T22:42:14.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5676016640 bytes 2026-02-20T22:42:14.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5737964544 bytes 2026-02-20T22:42:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5750105088 bytes 2026-02-20T22:42:14.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5761158144 bytes 2026-02-20T22:42:14.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5826845696 bytes 2026-02-20T22:42:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5908727808 bytes 2026-02-20T22:42:15.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5985912832 bytes 2026-02-20T22:42:15.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6026919936 bytes 2026-02-20T22:42:15.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6074284032 bytes 2026-02-20T22:42:15.136 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:42:15.126+0000 7f7ed1951640 -1 reset not still connected to 0x55c7097d3d40 2026-02-20T22:42:15.136 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:42:15.130+0000 7f7ed1951640 -1 reset not still connected to 0x55c70de580d0 2026-02-20T22:42:15.136 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:42:15.130+0000 7f7ed1951640 -1 reset not still connected to 0x55c737558c30 2026-02-20T22:42:15.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6154940416 bytes 2026-02-20T22:42:15.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6184554496 bytes 2026-02-20T22:42:15.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6213207040 bytes 2026-02-20T22:42:15.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6213838848 bytes 2026-02-20T22:42:15.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6221989888 bytes 2026-02-20T22:42:15.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6247067648 bytes 2026-02-20T22:42:15.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6317053952 bytes 2026-02-20T22:42:15.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6365680640 bytes 2026-02-20T22:42:15.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6382267392 bytes 2026-02-20T22:42:15.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6421028864 bytes 2026-02-20T22:42:15.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6483322880 bytes 2026-02-20T22:42:15.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6529636352 bytes 2026-02-20T22:42:15.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6573247488 bytes 2026-02-20T22:42:15.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6618378240 bytes 2026-02-20T22:42:15.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6655730688 bytes 2026-02-20T22:42:16.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6740449280 bytes 2026-02-20T22:42:16.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6782113792 bytes 2026-02-20T22:42:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6856214528 bytes 2026-02-20T22:42:16.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6895882240 bytes 2026-02-20T22:42:16.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6910702592 bytes 2026-02-20T22:42:16.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6964448256 bytes 2026-02-20T22:42:16.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6979827712 bytes 2026-02-20T22:42:16.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7000616960 bytes 2026-02-20T22:42:16.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7018129408 bytes 2026-02-20T22:42:16.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7046185984 bytes 2026-02-20T22:42:16.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7127439360 bytes 2026-02-20T22:42:16.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7140922368 bytes 2026-02-20T22:42:16.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7178058752 bytes 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_38 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2026-02-20T22:42:16.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_38 128M 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:16.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:16.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2026-02-20T22:42:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:16.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:16.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2026-02-20T22:42:16.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:16.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2026-02-20T22:42:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_38': 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1411fcc4ffdd 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1411fcc4ffdd 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:16 2026 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:16 2026 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:16 2026 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 12d9288b-1872-49ea-be06-e6e80ff2ebff 2026-02-20T22:42:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_38 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:42:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:42:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:17.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 31970304 bytes 2026-02-20T22:42:17.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 79794176 bytes 2026-02-20T22:42:17.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 159549440 bytes 2026-02-20T22:42:17.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 197276672 bytes 2026-02-20T22:42:17.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 232237056 bytes 2026-02-20T22:42:17.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 286873600 bytes 2026-02-20T22:42:17.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 339502080 bytes 2026-02-20T22:42:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 368210944 bytes 2026-02-20T22:42:17.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 404685824 bytes 2026-02-20T22:42:17.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 429280256 bytes 2026-02-20T22:42:17.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 476529664 bytes 2026-02-20T22:42:17.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 504726528 bytes 2026-02-20T22:42:17.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 567075840 bytes 2026-02-20T22:42:17.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 637450240 bytes 2026-02-20T22:42:17.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 708709376 bytes 2026-02-20T22:42:17.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 729174016 bytes 2026-02-20T22:42:17.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 753863680 bytes 2026-02-20T22:42:17.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 766120960 bytes 2026-02-20T22:42:18.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 769275904 bytes 2026-02-20T22:42:18.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 810360832 bytes 2026-02-20T22:42:18.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 854408192 bytes 2026-02-20T22:42:18.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 986477568 bytes 2026-02-20T22:42:18.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1003655168 bytes 2026-02-20T22:42:18.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1071819776 bytes 2026-02-20T22:42:18.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1139884032 bytes 2026-02-20T22:42:18.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1168626688 bytes 2026-02-20T22:42:18.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1223187456 bytes 2026-02-20T22:42:18.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1236835328 bytes 2026-02-20T22:42:18.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1278209024 bytes 2026-02-20T22:42:18.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1326224384 bytes 2026-02-20T22:42:18.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1339270144 bytes 2026-02-20T22:42:18.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1406807040 bytes 2026-02-20T22:42:18.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1407416320 bytes 2026-02-20T22:42:18.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1435858944 bytes 2026-02-20T22:42:19.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1499565056 bytes 2026-02-20T22:42:19.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1520754688 bytes 2026-02-20T22:42:19.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1566738432 bytes 2026-02-20T22:42:19.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1588168704 bytes 2026-02-20T22:42:19.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1611196416 bytes 2026-02-20T22:42:19.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1659493376 bytes 2026-02-20T22:42:19.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1701070848 bytes 2026-02-20T22:42:19.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1729817600 bytes 2026-02-20T22:42:19.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1756969984 bytes 2026-02-20T22:42:19.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1808832512 bytes 2026-02-20T22:42:19.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1864203264 bytes 2026-02-20T22:42:19.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1865824256 bytes 2026-02-20T22:42:19.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1866435584 bytes 2026-02-20T22:42:19.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1891282944 bytes 2026-02-20T22:42:19.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1960809472 bytes 2026-02-20T22:42:19.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1987625984 bytes 2026-02-20T22:42:19.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1996744704 bytes 2026-02-20T22:42:19.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2086210560 bytes 2026-02-20T22:42:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2113014784 bytes 2026-02-20T22:42:19.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2120952832 bytes 2026-02-20T22:42:20.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2176951296 bytes 2026-02-20T22:42:20.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2261137408 bytes 2026-02-20T22:42:20.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2311137280 bytes 2026-02-20T22:42:20.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2375485440 bytes 2026-02-20T22:42:20.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2386985984 bytes 2026-02-20T22:42:20.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2444840960 bytes 2026-02-20T22:42:20.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2445473792 bytes 2026-02-20T22:42:20.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2499303424 bytes 2026-02-20T22:42:20.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2575380480 bytes 2026-02-20T22:42:20.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2670935040 bytes 2026-02-20T22:42:20.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2723054592 bytes 2026-02-20T22:42:20.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2797128704 bytes 2026-02-20T22:42:20.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2853661696 bytes 2026-02-20T22:42:20.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2940417024 bytes 2026-02-20T22:42:20.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2979482624 bytes 2026-02-20T22:42:20.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2992429056 bytes 2026-02-20T22:42:21.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3034683392 bytes 2026-02-20T22:42:21.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3062012928 bytes 2026-02-20T22:42:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3119977472 bytes 2026-02-20T22:42:21.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3169262592 bytes 2026-02-20T22:42:21.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3188585472 bytes 2026-02-20T22:42:21.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3239209984 bytes 2026-02-20T22:42:21.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3298173952 bytes 2026-02-20T22:42:21.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3328406528 bytes 2026-02-20T22:42:21.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3369979904 bytes 2026-02-20T22:42:21.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3442043904 bytes 2026-02-20T22:42:21.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3490343936 bytes 2026-02-20T22:42:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3605568512 bytes 2026-02-20T22:42:21.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3665755136 bytes 2026-02-20T22:42:21.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3670592512 bytes 2026-02-20T22:42:21.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3715504128 bytes 2026-02-20T22:42:21.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3733602304 bytes 2026-02-20T22:42:21.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3757324288 bytes 2026-02-20T22:42:22.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3791242240 bytes 2026-02-20T22:42:22.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3795023872 bytes 2026-02-20T22:42:22.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3871649792 bytes 2026-02-20T22:42:22.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3911825408 bytes 2026-02-20T22:42:22.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3931672576 bytes 2026-02-20T22:42:22.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3977758720 bytes 2026-02-20T22:42:22.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4000054272 bytes 2026-02-20T22:42:22.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4021600256 bytes 2026-02-20T22:42:22.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4057808896 bytes 2026-02-20T22:42:22.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4082791424 bytes 2026-02-20T22:42:22.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4130548736 bytes 2026-02-20T22:42:22.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4226225152 bytes 2026-02-20T22:42:22.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4276434944 bytes 2026-02-20T22:42:22.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4277020672 bytes 2026-02-20T22:42:22.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4309296128 bytes 2026-02-20T22:42:22.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4342772736 bytes 2026-02-20T22:42:22.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4408288256 bytes 2026-02-20T22:42:22.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4453915648 bytes 2026-02-20T22:42:22.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4475284480 bytes 2026-02-20T22:42:23.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4536743936 bytes 2026-02-20T22:42:23.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4572533760 bytes 2026-02-20T22:42:23.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4603082752 bytes 2026-02-20T22:42:23.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4658821120 bytes 2026-02-20T22:42:23.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4696833024 bytes 2026-02-20T22:42:23.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4788516864 bytes 2026-02-20T22:42:23.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4818122752 bytes 2026-02-20T22:42:23.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4855676928 bytes 2026-02-20T22:42:23.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4931074048 bytes 2026-02-20T22:42:23.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4975657984 bytes 2026-02-20T22:42:23.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4990169088 bytes 2026-02-20T22:42:23.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5052380160 bytes 2026-02-20T22:42:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5052991488 bytes 2026-02-20T22:42:23.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5053619200 bytes 2026-02-20T22:42:23.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5094336512 bytes 2026-02-20T22:42:23.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5119059968 bytes 2026-02-20T22:42:23.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5198807040 bytes 2026-02-20T22:42:24.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5263536128 bytes 2026-02-20T22:42:24.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5268546560 bytes 2026-02-20T22:42:24.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5309093888 bytes 2026-02-20T22:42:24.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5332549632 bytes 2026-02-20T22:42:24.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5390898176 bytes 2026-02-20T22:42:24.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5434923008 bytes 2026-02-20T22:42:24.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5467996160 bytes 2026-02-20T22:42:24.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5568373760 bytes 2026-02-20T22:42:24.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5625781248 bytes 2026-02-20T22:42:24.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5688386560 bytes 2026-02-20T22:42:24.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5712929792 bytes 2026-02-20T22:42:24.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5765075968 bytes 2026-02-20T22:42:24.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5765683200 bytes 2026-02-20T22:42:24.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5885855744 bytes 2026-02-20T22:42:24.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5905622016 bytes 2026-02-20T22:42:24.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5927544832 bytes 2026-02-20T22:42:24.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5930183680 bytes 2026-02-20T22:42:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6000119808 bytes 2026-02-20T22:42:25.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6059376640 bytes 2026-02-20T22:42:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6092172288 bytes 2026-02-20T22:42:25.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6134175744 bytes 2026-02-20T22:42:25.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6146761728 bytes 2026-02-20T22:42:25.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6160100352 bytes 2026-02-20T22:42:25.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6167985152 bytes 2026-02-20T22:42:25.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6212736000 bytes 2026-02-20T22:42:25.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6219214848 bytes 2026-02-20T22:42:25.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6282914816 bytes 2026-02-20T22:42:25.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6316602368 bytes 2026-02-20T22:42:25.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6363466752 bytes 2026-02-20T22:42:25.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6438919168 bytes 2026-02-20T22:42:25.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6457762816 bytes 2026-02-20T22:42:25.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6474934272 bytes 2026-02-20T22:42:25.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6529815552 bytes 2026-02-20T22:42:25.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6558307328 bytes 2026-02-20T22:42:25.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6622151680 bytes 2026-02-20T22:42:26.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6669276160 bytes 2026-02-20T22:42:26.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6693146624 bytes 2026-02-20T22:42:26.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6730164224 bytes 2026-02-20T22:42:26.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6744470528 bytes 2026-02-20T22:42:26.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6808597504 bytes 2026-02-20T22:42:26.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6876711936 bytes 2026-02-20T22:42:26.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6934963200 bytes 2026-02-20T22:42:26.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6977904640 bytes 2026-02-20T22:42:26.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7013829632 bytes 2026-02-20T22:42:26.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7042342912 bytes 2026-02-20T22:42:26.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7100952576 bytes 2026-02-20T22:42:26.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7123053568 bytes 2026-02-20T22:42:26.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7130676224 bytes 2026-02-20T22:42:26.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7215515648 bytes 2026-02-20T22:42:26.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7276705792 bytes 2026-02-20T22:42:26.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7291970560 bytes 2026-02-20T22:42:26.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7292587008 bytes 2026-02-20T22:42:27.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7352955904 bytes 2026-02-20T22:42:27.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7406200832 bytes 2026-02-20T22:42:27.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7455401984 bytes 2026-02-20T22:42:27.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7467598848 bytes 2026-02-20T22:42:27.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7556624384 bytes 2026-02-20T22:42:27.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7623882752 bytes 2026-02-20T22:42:27.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7709475840 bytes 2026-02-20T22:42:27.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7721416704 bytes 2026-02-20T22:42:27.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7744852992 bytes 2026-02-20T22:42:27.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7758438400 bytes 2026-02-20T22:42:27.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7787279360 bytes 2026-02-20T22:42:27.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7874664448 bytes 2026-02-20T22:42:27.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7909305344 bytes 2026-02-20T22:42:27.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7923000320 bytes 2026-02-20T22:42:27.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7955479552 bytes 2026-02-20T22:42:27.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8042914816 bytes 2026-02-20T22:42:27.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8088015872 bytes 2026-02-20T22:42:28.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8149146624 bytes 2026-02-20T22:42:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8191295488 bytes 2026-02-20T22:42:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8191920128 bytes 2026-02-20T22:42:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8266246144 bytes 2026-02-20T22:42:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8294743040 bytes 2026-02-20T22:42:28.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8321693696 bytes 2026-02-20T22:42:28.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8330952704 bytes 2026-02-20T22:42:28.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8391823360 bytes 2026-02-20T22:42:28.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8436002816 bytes 2026-02-20T22:42:28.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8470571008 bytes 2026-02-20T22:42:28.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8508960768 bytes 2026-02-20T22:42:28.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8526994432 bytes 2026-02-20T22:42:28.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8531509248 bytes 2026-02-20T22:42:28.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8546872320 bytes 2026-02-20T22:42:28.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8569540608 bytes 2026-02-20T22:42:28.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8615101440 bytes 2026-02-20T22:42:28.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8655481856 bytes 2026-02-20T22:42:28.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8718000128 bytes 2026-02-20T22:42:28.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8794362880 bytes 2026-02-20T22:42:29.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 8813031424 bytes 2026-02-20T22:42:29.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 8894501888 bytes 2026-02-20T22:42:29.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 8917396480 bytes 2026-02-20T22:42:29.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 8972281856 bytes 2026-02-20T22:42:29.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 8972878848 bytes 2026-02-20T22:42:29.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9015943168 bytes 2026-02-20T22:42:29.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9072431104 bytes 2026-02-20T22:42:29.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9169376256 bytes 2026-02-20T22:42:29.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9196703744 bytes 2026-02-20T22:42:29.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9241379840 bytes 2026-02-20T22:42:29.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9317196800 bytes 2026-02-20T22:42:29.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9400737792 bytes 2026-02-20T22:42:29.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9421642752 bytes 2026-02-20T22:42:29.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9488016384 bytes 2026-02-20T22:42:29.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9541388288 bytes 2026-02-20T22:42:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:29.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_39 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:29.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_39 128M 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:29.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2026-02-20T22:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:29.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2026-02-20T22:42:30.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:30.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_39': 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 141b8f91c5f2 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.141b8f91c5f2 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:30.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:29 2026 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:29 2026 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:29 2026 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 403c7020-8ef5-46bd-81c8-d009d39aabd9 2026-02-20T22:42:30.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_39 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:42:30.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:30.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:31.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 64316416 bytes 2026-02-20T22:42:31.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 87521280 bytes 2026-02-20T22:42:31.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 157048832 bytes 2026-02-20T22:42:31.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 190273536 bytes 2026-02-20T22:42:31.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 202466304 bytes 2026-02-20T22:42:31.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 252983296 bytes 2026-02-20T22:42:31.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 270943232 bytes 2026-02-20T22:42:31.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 309099520 bytes 2026-02-20T22:42:31.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 355190784 bytes 2026-02-20T22:42:31.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 438674432 bytes 2026-02-20T22:42:31.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 507131904 bytes 2026-02-20T22:42:31.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 521683968 bytes 2026-02-20T22:42:31.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 557349888 bytes 2026-02-20T22:42:31.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 622279680 bytes 2026-02-20T22:42:31.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 672330752 bytes 2026-02-20T22:42:31.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 673785856 bytes 2026-02-20T22:42:31.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 711355392 bytes 2026-02-20T22:42:32.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 732044288 bytes 2026-02-20T22:42:32.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 779292672 bytes 2026-02-20T22:42:32.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 824263680 bytes 2026-02-20T22:42:32.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 863580160 bytes 2026-02-20T22:42:32.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 928384000 bytes 2026-02-20T22:42:32.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 958724096 bytes 2026-02-20T22:42:32.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1001505792 bytes 2026-02-20T22:42:32.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1009296384 bytes 2026-02-20T22:42:32.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1081763840 bytes 2026-02-20T22:42:32.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1119552512 bytes 2026-02-20T22:42:32.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1137676288 bytes 2026-02-20T22:42:32.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1182678016 bytes 2026-02-20T22:42:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1226842112 bytes 2026-02-20T22:42:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1260450816 bytes 2026-02-20T22:42:32.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1294351360 bytes 2026-02-20T22:42:32.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1294970880 bytes 2026-02-20T22:42:32.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1317754880 bytes 2026-02-20T22:42:32.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1368594432 bytes 2026-02-20T22:42:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1403540480 bytes 2026-02-20T22:42:33.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1416604672 bytes 2026-02-20T22:42:33.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1464881152 bytes 2026-02-20T22:42:33.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1549527040 bytes 2026-02-20T22:42:33.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1602563072 bytes 2026-02-20T22:42:33.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1651509248 bytes 2026-02-20T22:42:33.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1681679360 bytes 2026-02-20T22:42:33.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1696592896 bytes 2026-02-20T22:42:33.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1749675008 bytes 2026-02-20T22:42:33.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1771360256 bytes 2026-02-20T22:42:33.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1800708096 bytes 2026-02-20T22:42:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1828372480 bytes 2026-02-20T22:42:33.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1835596800 bytes 2026-02-20T22:42:33.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1862730752 bytes 2026-02-20T22:42:33.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1898869760 bytes 2026-02-20T22:42:33.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1958343680 bytes 2026-02-20T22:42:33.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1995827200 bytes 2026-02-20T22:42:33.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2024753152 bytes 2026-02-20T22:42:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2064891904 bytes 2026-02-20T22:42:34.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2133466112 bytes 2026-02-20T22:42:34.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2150319104 bytes 2026-02-20T22:42:34.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2248397824 bytes 2026-02-20T22:42:34.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2271901696 bytes 2026-02-20T22:42:34.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2298208256 bytes 2026-02-20T22:42:34.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2324498432 bytes 2026-02-20T22:42:34.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2372914176 bytes 2026-02-20T22:42:34.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2391736320 bytes 2026-02-20T22:42:34.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2451129344 bytes 2026-02-20T22:42:34.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2467688448 bytes 2026-02-20T22:42:34.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2485799936 bytes 2026-02-20T22:42:34.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2546814976 bytes 2026-02-20T22:42:34.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2621199360 bytes 2026-02-20T22:42:34.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2651401216 bytes 2026-02-20T22:42:34.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2666313728 bytes 2026-02-20T22:42:34.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2670627840 bytes 2026-02-20T22:42:35.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2766788608 bytes 2026-02-20T22:42:35.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2796014592 bytes 2026-02-20T22:42:35.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2835221504 bytes 2026-02-20T22:42:35.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2893265920 bytes 2026-02-20T22:42:35.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3008505856 bytes 2026-02-20T22:42:35.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3035459584 bytes 2026-02-20T22:42:35.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3052843008 bytes 2026-02-20T22:42:35.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3092547584 bytes 2026-02-20T22:42:35.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3169804288 bytes 2026-02-20T22:42:35.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3232770048 bytes 2026-02-20T22:42:35.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3275222016 bytes 2026-02-20T22:42:35.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3303482368 bytes 2026-02-20T22:42:35.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3347849216 bytes 2026-02-20T22:42:35.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3430926336 bytes 2026-02-20T22:42:35.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3493920768 bytes 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_40 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_40 128M 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2026-02-20T22:42:35.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:35.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:35.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:35.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:35.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2026-02-20T22:42:36.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:36.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_40': 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1424aa488427 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1424aa488427 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:35 2026 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:35 2026 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:35 2026 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2ce84f99-cd8f-4fec-ba8c-66a66d118bbc 2026-02-20T22:42:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_40 2026-02-20T22:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:42:36.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:36.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=9 2026-02-20T22:42:36.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:36.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:37.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 13824000 bytes 2026-02-20T22:42:37.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 86276096 bytes 2026-02-20T22:42:37.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 109810688 bytes 2026-02-20T22:42:37.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 146767872 bytes 2026-02-20T22:42:37.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 178050048 bytes 2026-02-20T22:42:37.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 226286592 bytes 2026-02-20T22:42:37.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 283686912 bytes 2026-02-20T22:42:37.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 293790720 bytes 2026-02-20T22:42:37.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 352930816 bytes 2026-02-20T22:42:37.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 369459200 bytes 2026-02-20T22:42:37.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 384026624 bytes 2026-02-20T22:42:37.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 469540864 bytes 2026-02-20T22:42:37.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 494781440 bytes 2026-02-20T22:42:37.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 543933440 bytes 2026-02-20T22:42:37.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 585211904 bytes 2026-02-20T22:42:37.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 609441792 bytes 2026-02-20T22:42:37.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 641186816 bytes 2026-02-20T22:42:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 760876032 bytes 2026-02-20T22:42:38.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 813290496 bytes 2026-02-20T22:42:38.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 823593984 bytes 2026-02-20T22:42:38.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 897723392 bytes 2026-02-20T22:42:38.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 940157952 bytes 2026-02-20T22:42:38.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1057827840 bytes 2026-02-20T22:42:38.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1058438144 bytes 2026-02-20T22:42:38.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1100726272 bytes 2026-02-20T22:42:38.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1150407680 bytes 2026-02-20T22:42:38.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1184405504 bytes 2026-02-20T22:42:38.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1209210880 bytes 2026-02-20T22:42:38.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1229626368 bytes 2026-02-20T22:42:38.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1278574592 bytes 2026-02-20T22:42:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1310931968 bytes 2026-02-20T22:42:38.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1389306880 bytes 2026-02-20T22:42:38.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1442682880 bytes 2026-02-20T22:42:38.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1488748544 bytes 2026-02-20T22:42:39.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1565952000 bytes 2026-02-20T22:42:39.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1596675072 bytes 2026-02-20T22:42:39.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1682268160 bytes 2026-02-20T22:42:39.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1729442816 bytes 2026-02-20T22:42:39.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1760408576 bytes 2026-02-20T22:42:39.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1803979776 bytes 2026-02-20T22:42:39.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1895426048 bytes 2026-02-20T22:42:39.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1939045376 bytes 2026-02-20T22:42:39.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1978188800 bytes 2026-02-20T22:42:39.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2012311552 bytes 2026-02-20T22:42:39.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2065649664 bytes 2026-02-20T22:42:39.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2096538624 bytes 2026-02-20T22:42:39.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2140075008 bytes 2026-02-20T22:42:39.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2140658688 bytes 2026-02-20T22:42:39.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2172989440 bytes 2026-02-20T22:42:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2208623616 bytes 2026-02-20T22:42:39.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2226793472 bytes 2026-02-20T22:42:40.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2317375488 bytes 2026-02-20T22:42:40.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2330977280 bytes 2026-02-20T22:42:40.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2373935104 bytes 2026-02-20T22:42:40.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2410164224 bytes 2026-02-20T22:42:40.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2454254592 bytes 2026-02-20T22:42:40.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2475618304 bytes 2026-02-20T22:42:40.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2555573248 bytes 2026-02-20T22:42:40.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2642956288 bytes 2026-02-20T22:42:40.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2677264384 bytes 2026-02-20T22:42:40.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2696052736 bytes 2026-02-20T22:42:40.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2762508288 bytes 2026-02-20T22:42:40.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2851087360 bytes 2026-02-20T22:42:40.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2876763136 bytes 2026-02-20T22:42:40.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2964168704 bytes 2026-02-20T22:42:40.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2990796800 bytes 2026-02-20T22:42:41.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3042340864 bytes 2026-02-20T22:42:41.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3115971584 bytes 2026-02-20T22:42:41.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3142402048 bytes 2026-02-20T22:42:41.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3184897024 bytes 2026-02-20T22:42:41.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3257398272 bytes 2026-02-20T22:42:41.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3317543936 bytes 2026-02-20T22:42:41.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3328881664 bytes 2026-02-20T22:42:41.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3352529920 bytes 2026-02-20T22:42:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3402622976 bytes 2026-02-20T22:42:41.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3454394368 bytes 2026-02-20T22:42:41.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3482781696 bytes 2026-02-20T22:42:41.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3521876992 bytes 2026-02-20T22:42:41.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3532730368 bytes 2026-02-20T22:42:41.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3556079616 bytes 2026-02-20T22:42:41.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3608943616 bytes 2026-02-20T22:42:41.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3649160192 bytes 2026-02-20T22:42:41.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3670584320 bytes 2026-02-20T22:42:41.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3772453888 bytes 2026-02-20T22:42:42.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3794045952 bytes 2026-02-20T22:42:42.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3863767040 bytes 2026-02-20T22:42:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3893065728 bytes 2026-02-20T22:42:42.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3965345792 bytes 2026-02-20T22:42:42.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3999762432 bytes 2026-02-20T22:42:42.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4016232448 bytes 2026-02-20T22:42:42.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4052746240 bytes 2026-02-20T22:42:42.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4053392384 bytes 2026-02-20T22:42:42.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4054028288 bytes 2026-02-20T22:42:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4100908032 bytes 2026-02-20T22:42:42.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4141947904 bytes 2026-02-20T22:42:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4218488832 bytes 2026-02-20T22:42:42.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4239034368 bytes 2026-02-20T22:42:42.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4302920704 bytes 2026-02-20T22:42:42.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4351800320 bytes 2026-02-20T22:42:42.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4467974144 bytes 2026-02-20T22:42:42.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4516703232 bytes 2026-02-20T22:42:43.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4564417536 bytes 2026-02-20T22:42:43.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4633897984 bytes 2026-02-20T22:42:43.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4691476480 bytes 2026-02-20T22:42:43.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4727745536 bytes 2026-02-20T22:42:43.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4752828416 bytes 2026-02-20T22:42:43.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4781175808 bytes 2026-02-20T22:42:43.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4819017728 bytes 2026-02-20T22:42:43.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4850539520 bytes 2026-02-20T22:42:43.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4928498688 bytes 2026-02-20T22:42:43.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4978588672 bytes 2026-02-20T22:42:43.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5014879232 bytes 2026-02-20T22:42:43.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5047279616 bytes 2026-02-20T22:42:43.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5089435648 bytes 2026-02-20T22:42:43.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5129926656 bytes 2026-02-20T22:42:43.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5226130432 bytes 2026-02-20T22:42:43.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5264400384 bytes 2026-02-20T22:42:44.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5309450240 bytes 2026-02-20T22:42:44.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5334583296 bytes 2026-02-20T22:42:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5376583680 bytes 2026-02-20T22:42:44.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5486997504 bytes 2026-02-20T22:42:44.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5556501504 bytes 2026-02-20T22:42:44.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5588799488 bytes 2026-02-20T22:42:44.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5589427200 bytes 2026-02-20T22:42:44.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5651293184 bytes 2026-02-20T22:42:44.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5691797504 bytes 2026-02-20T22:42:44.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5722972160 bytes 2026-02-20T22:42:44.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5746001920 bytes 2026-02-20T22:42:44.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5758138368 bytes 2026-02-20T22:42:44.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5859730432 bytes 2026-02-20T22:42:45.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5879643136 bytes 2026-02-20T22:42:45.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5971554304 bytes 2026-02-20T22:42:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6007438336 bytes 2026-02-20T22:42:45.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6053720064 bytes 2026-02-20T22:42:45.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6069601280 bytes 2026-02-20T22:42:45.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6118133760 bytes 2026-02-20T22:42:45.404 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6161345536 bytes 2026-02-20T22:42:45.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6168842240 bytes 2026-02-20T22:42:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6230711296 bytes 2026-02-20T22:42:45.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6265302016 bytes 2026-02-20T22:42:45.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6317203456 bytes 2026-02-20T22:42:45.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6345716736 bytes 2026-02-20T22:42:45.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6346962944 bytes 2026-02-20T22:42:45.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6377771008 bytes 2026-02-20T22:42:45.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6432704512 bytes 2026-02-20T22:42:45.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6494012416 bytes 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_41 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2026-02-20T22:42:45.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_41 128M 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:45.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2026-02-20T22:42:45.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:45.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:45.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:45.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2026-02-20T22:42:46.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:46.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2026-02-20T22:42:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_41': 2026-02-20T22:42:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 142e379f7b40 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.142e379f7b40 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:45 2026 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:45 2026 2026-02-20T22:42:46.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:45 2026 2026-02-20T22:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 73979901-0e9a-4163-876a-7c36d69ac17d 2026-02-20T22:42:46.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_41 2026-02-20T22:42:46.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:42:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:42:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:46.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 8560640 bytes 2026-02-20T22:42:46.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 54331392 bytes 2026-02-20T22:42:46.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 76004352 bytes 2026-02-20T22:42:46.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 105771008 bytes 2026-02-20T22:42:46.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 162151424 bytes 2026-02-20T22:42:47.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 212063232 bytes 2026-02-20T22:42:47.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 230661120 bytes 2026-02-20T22:42:47.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 278023168 bytes 2026-02-20T22:42:47.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 320012288 bytes 2026-02-20T22:42:47.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 345294848 bytes 2026-02-20T22:42:47.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 393604096 bytes 2026-02-20T22:42:47.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 449133568 bytes 2026-02-20T22:42:47.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 462494720 bytes 2026-02-20T22:42:47.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 551257088 bytes 2026-02-20T22:42:47.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 619111424 bytes 2026-02-20T22:42:47.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 640279552 bytes 2026-02-20T22:42:47.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 691057664 bytes 2026-02-20T22:42:47.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 758583296 bytes 2026-02-20T22:42:47.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 760973312 bytes 2026-02-20T22:42:47.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 779481088 bytes 2026-02-20T22:42:47.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 839406592 bytes 2026-02-20T22:42:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 881558528 bytes 2026-02-20T22:42:47.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 916199424 bytes 2026-02-20T22:42:48.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 949921792 bytes 2026-02-20T22:42:48.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1015913472 bytes 2026-02-20T22:42:48.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1102610432 bytes 2026-02-20T22:42:48.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1107838976 bytes 2026-02-20T22:42:48.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1146458112 bytes 2026-02-20T22:42:48.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1200313344 bytes 2026-02-20T22:42:48.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1229315072 bytes 2026-02-20T22:42:48.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1243882496 bytes 2026-02-20T22:42:48.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1244497920 bytes 2026-02-20T22:42:48.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1261969408 bytes 2026-02-20T22:42:48.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1311245312 bytes 2026-02-20T22:42:48.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1341844480 bytes 2026-02-20T22:42:48.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1379629056 bytes 2026-02-20T22:42:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1445517312 bytes 2026-02-20T22:42:48.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1461416960 bytes 2026-02-20T22:42:48.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1476326400 bytes 2026-02-20T22:42:48.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1523312640 bytes 2026-02-20T22:42:48.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1604214784 bytes 2026-02-20T22:42:49.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1696414720 bytes 2026-02-20T22:42:49.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1790225408 bytes 2026-02-20T22:42:49.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1818653696 bytes 2026-02-20T22:42:49.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1828137984 bytes 2026-02-20T22:42:49.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1866701824 bytes 2026-02-20T22:42:49.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1923025920 bytes 2026-02-20T22:42:49.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1961850880 bytes 2026-02-20T22:42:49.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2027123712 bytes 2026-02-20T22:42:49.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2074127360 bytes 2026-02-20T22:42:49.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2089853952 bytes 2026-02-20T22:42:49.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2184196096 bytes 2026-02-20T22:42:49.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2214546432 bytes 2026-02-20T22:42:49.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2269417472 bytes 2026-02-20T22:42:49.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2307648512 bytes 2026-02-20T22:42:49.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2334412800 bytes 2026-02-20T22:42:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2372692992 bytes 2026-02-20T22:42:49.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2391994368 bytes 2026-02-20T22:42:50.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2445485056 bytes 2026-02-20T22:42:50.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2489276416 bytes 2026-02-20T22:42:50.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2545030144 bytes 2026-02-20T22:42:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2607396864 bytes 2026-02-20T22:42:50.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2663657472 bytes 2026-02-20T22:42:50.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2707180544 bytes 2026-02-20T22:42:50.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2743241728 bytes 2026-02-20T22:42:50.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2777269248 bytes 2026-02-20T22:42:50.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2803237888 bytes 2026-02-20T22:42:50.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2834981888 bytes 2026-02-20T22:42:50.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2847894528 bytes 2026-02-20T22:42:50.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2860720128 bytes 2026-02-20T22:42:50.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2891885568 bytes 2026-02-20T22:42:50.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2946016256 bytes 2026-02-20T22:42:50.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3010419712 bytes 2026-02-20T22:42:50.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3047198720 bytes 2026-02-20T22:42:50.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3080239104 bytes 2026-02-20T22:42:50.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3124253696 bytes 2026-02-20T22:42:51.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3128270848 bytes 2026-02-20T22:42:51.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3161267200 bytes 2026-02-20T22:42:51.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3170889728 bytes 2026-02-20T22:42:51.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3207452672 bytes 2026-02-20T22:42:51.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3280717824 bytes 2026-02-20T22:42:51.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3314198528 bytes 2026-02-20T22:42:51.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3371660288 bytes 2026-02-20T22:42:51.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3401938944 bytes 2026-02-20T22:42:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3431448576 bytes 2026-02-20T22:42:51.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3453388800 bytes 2026-02-20T22:42:51.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3470967808 bytes 2026-02-20T22:42:51.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3499634688 bytes 2026-02-20T22:42:51.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3553910784 bytes 2026-02-20T22:42:51.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3573512192 bytes 2026-02-20T22:42:51.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3613034496 bytes 2026-02-20T22:42:51.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3619616768 bytes 2026-02-20T22:42:51.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3627881472 bytes 2026-02-20T22:42:51.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3689971712 bytes 2026-02-20T22:42:51.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3747961856 bytes 2026-02-20T22:42:51.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3784037376 bytes 2026-02-20T22:42:52.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3841228800 bytes 2026-02-20T22:42:52.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3888358400 bytes 2026-02-20T22:42:52.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3957648384 bytes 2026-02-20T22:42:52.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3993994240 bytes 2026-02-20T22:42:52.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4050994176 bytes 2026-02-20T22:42:52.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4118470656 bytes 2026-02-20T22:42:52.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4162512896 bytes 2026-02-20T22:42:52.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4271795200 bytes 2026-02-20T22:42:52.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4283934720 bytes 2026-02-20T22:42:52.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4347451392 bytes 2026-02-20T22:42:52.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4362807296 bytes 2026-02-20T22:42:52.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4418130944 bytes 2026-02-20T22:42:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4431444992 bytes 2026-02-20T22:42:52.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4474169344 bytes 2026-02-20T22:42:52.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4547296256 bytes 2026-02-20T22:42:52.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4565646336 bytes 2026-02-20T22:42:52.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4600105984 bytes 2026-02-20T22:42:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4654213120 bytes 2026-02-20T22:42:53.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4660865024 bytes 2026-02-20T22:42:53.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4732358656 bytes 2026-02-20T22:42:53.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4774273024 bytes 2026-02-20T22:42:53.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4791587840 bytes 2026-02-20T22:42:53.335 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4824801280 bytes 2026-02-20T22:42:53.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4896502784 bytes 2026-02-20T22:42:53.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5006287872 bytes 2026-02-20T22:42:53.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5079284736 bytes 2026-02-20T22:42:53.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5128112128 bytes 2026-02-20T22:42:53.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_42 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_42 128M 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:53.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:53.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:53.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:53.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2026-02-20T22:42:53.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:53.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2026-02-20T22:42:54.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:42:54.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2026-02-20T22:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_42': 2026-02-20T22:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:42:54.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1436b45de8e1 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1436b45de8e1 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:53 2026 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:53 2026 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:53 2026 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3b9366c3-0cd7-4131-958f-35709e2da94a 2026-02-20T22:42:54.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:42:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:42:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_42 2026-02-20T22:42:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:54.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:54.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:42:54.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:42:54.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:42:54.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:42:54.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:42:54.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 15420416 bytes 2026-02-20T22:42:54.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 46141440 bytes 2026-02-20T22:42:54.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 104758272 bytes 2026-02-20T22:42:54.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 124461056 bytes 2026-02-20T22:42:54.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 148419584 bytes 2026-02-20T22:42:55.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 160001024 bytes 2026-02-20T22:42:55.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 229214208 bytes 2026-02-20T22:42:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 282534912 bytes 2026-02-20T22:42:55.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 313825280 bytes 2026-02-20T22:42:55.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 332817408 bytes 2026-02-20T22:42:55.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 373951488 bytes 2026-02-20T22:42:55.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 416835584 bytes 2026-02-20T22:42:55.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 458199040 bytes 2026-02-20T22:42:55.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 482892800 bytes 2026-02-20T22:42:55.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 517074944 bytes 2026-02-20T22:42:55.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 570983424 bytes 2026-02-20T22:42:55.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 640769024 bytes 2026-02-20T22:42:55.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 684827648 bytes 2026-02-20T22:42:55.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 700034048 bytes 2026-02-20T22:42:55.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 730940416 bytes 2026-02-20T22:42:55.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 821437440 bytes 2026-02-20T22:42:55.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 882535424 bytes 2026-02-20T22:42:56.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 912376832 bytes 2026-02-20T22:42:56.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 947283968 bytes 2026-02-20T22:42:56.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1050554368 bytes 2026-02-20T22:42:56.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1070960640 bytes 2026-02-20T22:42:56.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1140340736 bytes 2026-02-20T22:42:56.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1177937920 bytes 2026-02-20T22:42:56.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1217457152 bytes 2026-02-20T22:42:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1237068800 bytes 2026-02-20T22:42:56.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1282073600 bytes 2026-02-20T22:42:56.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1316992000 bytes 2026-02-20T22:42:56.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1372766208 bytes 2026-02-20T22:42:56.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1395751936 bytes 2026-02-20T22:42:56.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1420045312 bytes 2026-02-20T22:42:56.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1435064320 bytes 2026-02-20T22:42:56.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1458906112 bytes 2026-02-20T22:42:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1468716032 bytes 2026-02-20T22:42:56.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1479148544 bytes 2026-02-20T22:42:56.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1572469760 bytes 2026-02-20T22:42:57.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1610641408 bytes 2026-02-20T22:42:57.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1632819200 bytes 2026-02-20T22:42:57.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1702170624 bytes 2026-02-20T22:42:57.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1757915136 bytes 2026-02-20T22:42:57.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1787458560 bytes 2026-02-20T22:42:57.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1837814784 bytes 2026-02-20T22:42:57.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1873677312 bytes 2026-02-20T22:42:57.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1902083072 bytes 2026-02-20T22:42:57.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1957227520 bytes 2026-02-20T22:42:57.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2006046720 bytes 2026-02-20T22:42:57.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2075500544 bytes 2026-02-20T22:42:57.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2100569088 bytes 2026-02-20T22:42:57.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2162829312 bytes 2026-02-20T22:42:57.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2176651264 bytes 2026-02-20T22:42:57.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2234641408 bytes 2026-02-20T22:42:57.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2301750272 bytes 2026-02-20T22:42:57.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2315600896 bytes 2026-02-20T22:42:58.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2328192000 bytes 2026-02-20T22:42:58.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2404658176 bytes 2026-02-20T22:42:58.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2448353280 bytes 2026-02-20T22:42:58.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2530381824 bytes 2026-02-20T22:42:58.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2552971264 bytes 2026-02-20T22:42:58.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2565937152 bytes 2026-02-20T22:42:58.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2601917440 bytes 2026-02-20T22:42:58.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2607538176 bytes 2026-02-20T22:42:58.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2623706112 bytes 2026-02-20T22:42:58.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2678197248 bytes 2026-02-20T22:42:58.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2723549184 bytes 2026-02-20T22:42:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2747444224 bytes 2026-02-20T22:42:58.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2767746048 bytes 2026-02-20T22:42:58.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2790258688 bytes 2026-02-20T22:42:58.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2831783936 bytes 2026-02-20T22:42:58.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2920929280 bytes 2026-02-20T22:42:58.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2978530304 bytes 2026-02-20T22:42:58.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3040239616 bytes 2026-02-20T22:42:59.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3090114560 bytes 2026-02-20T22:42:59.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3155337216 bytes 2026-02-20T22:42:59.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3248131072 bytes 2026-02-20T22:42:59.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3321338880 bytes 2026-02-20T22:42:59.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3394993152 bytes 2026-02-20T22:42:59.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3455882240 bytes 2026-02-20T22:42:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3480173568 bytes 2026-02-20T22:42:59.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3517341696 bytes 2026-02-20T22:42:59.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3544012800 bytes 2026-02-20T22:42:59.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3615251456 bytes 2026-02-20T22:42:59.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3626637312 bytes 2026-02-20T22:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:42:59.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_43 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:42:59.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_43 128M 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:42:59.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2026-02-20T22:42:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:42:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2026-02-20T22:43:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:00.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_43': 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1440ecc895e1 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1440ecc895e1 2026-02-20T22:43:00.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:59 2026 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:59 2026 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:59 2026 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ba45bb6e-d551-4afb-a2c1-3248377995b6 2026-02-20T22:43:00.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_43 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:43:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=9 2026-02-20T22:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 4545536 bytes 2026-02-20T22:43:00.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 46749696 bytes 2026-02-20T22:43:00.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 95226880 bytes 2026-02-20T22:43:00.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 157780992 bytes 2026-02-20T22:43:01.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 173000704 bytes 2026-02-20T22:43:01.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 204939264 bytes 2026-02-20T22:43:01.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 259404800 bytes 2026-02-20T22:43:01.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 357024768 bytes 2026-02-20T22:43:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 400850944 bytes 2026-02-20T22:43:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 432915456 bytes 2026-02-20T22:43:01.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 456530944 bytes 2026-02-20T22:43:01.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 495483904 bytes 2026-02-20T22:43:01.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 540083200 bytes 2026-02-20T22:43:01.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 566409216 bytes 2026-02-20T22:43:01.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 604066816 bytes 2026-02-20T22:43:01.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 765376512 bytes 2026-02-20T22:43:01.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 789744640 bytes 2026-02-20T22:43:01.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 864614400 bytes 2026-02-20T22:43:01.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 873290752 bytes 2026-02-20T22:43:01.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 904030208 bytes 2026-02-20T22:43:01.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 942816256 bytes 2026-02-20T22:43:01.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 943436800 bytes 2026-02-20T22:43:02.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 976462848 bytes 2026-02-20T22:43:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1055075328 bytes 2026-02-20T22:43:02.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1077008384 bytes 2026-02-20T22:43:02.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1108307968 bytes 2026-02-20T22:43:02.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1165517824 bytes 2026-02-20T22:43:02.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1213281280 bytes 2026-02-20T22:43:02.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1257609216 bytes 2026-02-20T22:43:02.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1362445312 bytes 2026-02-20T22:43:02.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1419649024 bytes 2026-02-20T22:43:02.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1435897856 bytes 2026-02-20T22:43:02.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1494286336 bytes 2026-02-20T22:43:02.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1533443072 bytes 2026-02-20T22:43:02.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1534049280 bytes 2026-02-20T22:43:02.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1601566720 bytes 2026-02-20T22:43:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1621881856 bytes 2026-02-20T22:43:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1692542976 bytes 2026-02-20T22:43:02.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1747309568 bytes 2026-02-20T22:43:03.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1870461952 bytes 2026-02-20T22:43:03.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1898306560 bytes 2026-02-20T22:43:03.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1898927104 bytes 2026-02-20T22:43:03.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1982324736 bytes 2026-02-20T22:43:03.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2088930304 bytes 2026-02-20T22:43:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2163470336 bytes 2026-02-20T22:43:03.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2180931584 bytes 2026-02-20T22:43:03.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2207442944 bytes 2026-02-20T22:43:03.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2253400064 bytes 2026-02-20T22:43:03.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2328947712 bytes 2026-02-20T22:43:03.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2401429504 bytes 2026-02-20T22:43:03.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2427008000 bytes 2026-02-20T22:43:03.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2459373568 bytes 2026-02-20T22:43:03.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2491761664 bytes 2026-02-20T22:43:03.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2496373760 bytes 2026-02-20T22:43:03.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2537455616 bytes 2026-02-20T22:43:04.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2616688640 bytes 2026-02-20T22:43:04.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2636972032 bytes 2026-02-20T22:43:04.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2722286592 bytes 2026-02-20T22:43:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2730916864 bytes 2026-02-20T22:43:04.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2754244608 bytes 2026-02-20T22:43:04.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2802450432 bytes 2026-02-20T22:43:04.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2910465024 bytes 2026-02-20T22:43:04.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2918971392 bytes 2026-02-20T22:43:04.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2953651200 bytes 2026-02-20T22:43:04.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2987735040 bytes 2026-02-20T22:43:04.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3044549632 bytes 2026-02-20T22:43:04.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3121664000 bytes 2026-02-20T22:43:04.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3160534016 bytes 2026-02-20T22:43:04.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3182271488 bytes 2026-02-20T22:43:04.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3182903296 bytes 2026-02-20T22:43:04.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3297038336 bytes 2026-02-20T22:43:05.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3382899712 bytes 2026-02-20T22:43:05.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3464128512 bytes 2026-02-20T22:43:05.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3497697280 bytes 2026-02-20T22:43:05.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3526230016 bytes 2026-02-20T22:43:05.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3577212928 bytes 2026-02-20T22:43:05.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3678276608 bytes 2026-02-20T22:43:05.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3717332992 bytes 2026-02-20T22:43:05.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3778481152 bytes 2026-02-20T22:43:05.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3819568128 bytes 2026-02-20T22:43:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3897099264 bytes 2026-02-20T22:43:05.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3943248896 bytes 2026-02-20T22:43:05.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3955310592 bytes 2026-02-20T22:43:05.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 4019765248 bytes 2026-02-20T22:43:05.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4069285888 bytes 2026-02-20T22:43:05.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4083075072 bytes 2026-02-20T22:43:05.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4113409024 bytes 2026-02-20T22:43:05.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4137461760 bytes 2026-02-20T22:43:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4220021760 bytes 2026-02-20T22:43:06.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4320629760 bytes 2026-02-20T22:43:06.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4392390656 bytes 2026-02-20T22:43:06.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4462291968 bytes 2026-02-20T22:43:06.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4515450880 bytes 2026-02-20T22:43:06.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4538770432 bytes 2026-02-20T22:43:06.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4574949376 bytes 2026-02-20T22:43:06.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4621650944 bytes 2026-02-20T22:43:06.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4693907456 bytes 2026-02-20T22:43:06.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4730295296 bytes 2026-02-20T22:43:06.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4778098688 bytes 2026-02-20T22:43:06.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4810001408 bytes 2026-02-20T22:43:06.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4830494720 bytes 2026-02-20T22:43:06.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4846535680 bytes 2026-02-20T22:43:06.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4859075584 bytes 2026-02-20T22:43:06.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4898938880 bytes 2026-02-20T22:43:06.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4953368576 bytes 2026-02-20T22:43:06.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4953980928 bytes 2026-02-20T22:43:07.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4955073536 bytes 2026-02-20T22:43:07.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4955678720 bytes 2026-02-20T22:43:07.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 5006171136 bytes 2026-02-20T22:43:07.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5033791488 bytes 2026-02-20T22:43:07.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5137750016 bytes 2026-02-20T22:43:07.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5223706624 bytes 2026-02-20T22:43:07.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5260214272 bytes 2026-02-20T22:43:07.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5287734272 bytes 2026-02-20T22:43:07.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5330825216 bytes 2026-02-20T22:43:07.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5392531456 bytes 2026-02-20T22:43:07.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5437910016 bytes 2026-02-20T22:43:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5542576128 bytes 2026-02-20T22:43:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5635822592 bytes 2026-02-20T22:43:07.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5667466240 bytes 2026-02-20T22:43:07.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5684583424 bytes 2026-02-20T22:43:07.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5748539392 bytes 2026-02-20T22:43:08.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5771986944 bytes 2026-02-20T22:43:08.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5787416576 bytes 2026-02-20T22:43:08.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5826592768 bytes 2026-02-20T22:43:08.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5849866240 bytes 2026-02-20T22:43:08.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5873798144 bytes 2026-02-20T22:43:08.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5895390208 bytes 2026-02-20T22:43:08.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5952456704 bytes 2026-02-20T22:43:08.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6033984512 bytes 2026-02-20T22:43:08.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6052670464 bytes 2026-02-20T22:43:08.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6097269760 bytes 2026-02-20T22:43:08.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6136760320 bytes 2026-02-20T22:43:08.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6224223232 bytes 2026-02-20T22:43:08.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6296394752 bytes 2026-02-20T22:43:08.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6304562176 bytes 2026-02-20T22:43:08.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6372318208 bytes 2026-02-20T22:43:08.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6380411904 bytes 2026-02-20T22:43:08.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6475251712 bytes 2026-02-20T22:43:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6475878400 bytes 2026-02-20T22:43:09.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6498451456 bytes 2026-02-20T22:43:09.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6519701504 bytes 2026-02-20T22:43:09.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6583670784 bytes 2026-02-20T22:43:09.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6600374272 bytes 2026-02-20T22:43:09.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6620546048 bytes 2026-02-20T22:43:09.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6663520256 bytes 2026-02-20T22:43:09.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6722622464 bytes 2026-02-20T22:43:09.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6820224000 bytes 2026-02-20T22:43:09.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6911216640 bytes 2026-02-20T22:43:09.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6934923264 bytes 2026-02-20T22:43:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6990815232 bytes 2026-02-20T22:43:09.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 7063331840 bytes 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_44 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_44 128M 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:09.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2026-02-20T22:43:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:43:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:43:09.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:09.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2026-02-20T22:43:10.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:10.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2026-02-20T22:43:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_44': 2026-02-20T22:43:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:10.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1449e942df28 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1449e942df28 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:09 2026 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:09 2026 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:09 2026 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:10.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:10.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9e589007-be33-45e3-a2f7-22c31b17078f 2026-02-20T22:43:10.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_44 2026-02-20T22:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:10.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:43:10.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:10.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:43:10.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:10.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:10.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 30493696 bytes 2026-02-20T22:43:10.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 69263360 bytes 2026-02-20T22:43:10.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 76300288 bytes 2026-02-20T22:43:10.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 76904448 bytes 2026-02-20T22:43:10.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 111854592 bytes 2026-02-20T22:43:11.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 127296512 bytes 2026-02-20T22:43:11.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 220135424 bytes 2026-02-20T22:43:11.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 259357696 bytes 2026-02-20T22:43:11.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 266005504 bytes 2026-02-20T22:43:11.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 316141568 bytes 2026-02-20T22:43:11.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 337744896 bytes 2026-02-20T22:43:11.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 355019776 bytes 2026-02-20T22:43:11.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 378938368 bytes 2026-02-20T22:43:11.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 466673664 bytes 2026-02-20T22:43:11.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 553633792 bytes 2026-02-20T22:43:11.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 614579200 bytes 2026-02-20T22:43:11.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 683895808 bytes 2026-02-20T22:43:11.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 717573120 bytes 2026-02-20T22:43:11.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 741001216 bytes 2026-02-20T22:43:11.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 814247936 bytes 2026-02-20T22:43:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 834814976 bytes 2026-02-20T22:43:11.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 867090432 bytes 2026-02-20T22:43:12.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 921481216 bytes 2026-02-20T22:43:12.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 930891776 bytes 2026-02-20T22:43:12.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 950730752 bytes 2026-02-20T22:43:12.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 994212864 bytes 2026-02-20T22:43:12.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1045155840 bytes 2026-02-20T22:43:12.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1072922624 bytes 2026-02-20T22:43:12.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1099348992 bytes 2026-02-20T22:43:12.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1123941376 bytes 2026-02-20T22:43:12.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1170483200 bytes 2026-02-20T22:43:12.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1210405888 bytes 2026-02-20T22:43:12.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1264990208 bytes 2026-02-20T22:43:12.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1385776128 bytes 2026-02-20T22:43:12.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1414597632 bytes 2026-02-20T22:43:12.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1453366272 bytes 2026-02-20T22:43:12.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1468789760 bytes 2026-02-20T22:43:12.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1506435072 bytes 2026-02-20T22:43:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1554038784 bytes 2026-02-20T22:43:12.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1634267136 bytes 2026-02-20T22:43:13.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1712617472 bytes 2026-02-20T22:43:13.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1800676352 bytes 2026-02-20T22:43:13.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1852694528 bytes 2026-02-20T22:43:13.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1908267008 bytes 2026-02-20T22:43:13.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2002406400 bytes 2026-02-20T22:43:13.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2067387392 bytes 2026-02-20T22:43:13.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2096103424 bytes 2026-02-20T22:43:13.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2103367680 bytes 2026-02-20T22:43:13.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2171847680 bytes 2026-02-20T22:43:13.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2209736704 bytes 2026-02-20T22:43:13.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2249015296 bytes 2026-02-20T22:43:13.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2305324032 bytes 2026-02-20T22:43:13.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2342717440 bytes 2026-02-20T22:43:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2451704832 bytes 2026-02-20T22:43:13.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2515262464 bytes 2026-02-20T22:43:14.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2542611456 bytes 2026-02-20T22:43:14.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2562809856 bytes 2026-02-20T22:43:14.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2632272896 bytes 2026-02-20T22:43:14.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2673946624 bytes 2026-02-20T22:43:14.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2731249664 bytes 2026-02-20T22:43:14.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2783409152 bytes 2026-02-20T22:43:14.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2821709824 bytes 2026-02-20T22:43:14.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2875473920 bytes 2026-02-20T22:43:14.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2913977344 bytes 2026-02-20T22:43:14.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2957177856 bytes 2026-02-20T22:43:14.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3002772480 bytes 2026-02-20T22:43:14.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3066627072 bytes 2026-02-20T22:43:14.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3094796288 bytes 2026-02-20T22:43:14.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3140450304 bytes 2026-02-20T22:43:14.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3153619968 bytes 2026-02-20T22:43:14.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3176018944 bytes 2026-02-20T22:43:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3274560512 bytes 2026-02-20T22:43:15.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3343463424 bytes 2026-02-20T22:43:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3398676480 bytes 2026-02-20T22:43:15.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3451885568 bytes 2026-02-20T22:43:15.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3460151296 bytes 2026-02-20T22:43:15.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3552247808 bytes 2026-02-20T22:43:15.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3614574592 bytes 2026-02-20T22:43:15.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3636955136 bytes 2026-02-20T22:43:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3646680064 bytes 2026-02-20T22:43:15.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3647967232 bytes 2026-02-20T22:43:15.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3736468480 bytes 2026-02-20T22:43:15.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3787708416 bytes 2026-02-20T22:43:15.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3812656128 bytes 2026-02-20T22:43:15.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3859024896 bytes 2026-02-20T22:43:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3876166656 bytes 2026-02-20T22:43:15.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3897590784 bytes 2026-02-20T22:43:15.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3989982208 bytes 2026-02-20T22:43:15.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4010703872 bytes 2026-02-20T22:43:15.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4066870272 bytes 2026-02-20T22:43:16.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4092530688 bytes 2026-02-20T22:43:16.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4142283776 bytes 2026-02-20T22:43:16.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4190671872 bytes 2026-02-20T22:43:16.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4255633408 bytes 2026-02-20T22:43:16.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4321829888 bytes 2026-02-20T22:43:16.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4346530816 bytes 2026-02-20T22:43:16.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4422417408 bytes 2026-02-20T22:43:16.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4463258624 bytes 2026-02-20T22:43:16.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4504476672 bytes 2026-02-20T22:43:16.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4582635520 bytes 2026-02-20T22:43:16.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4583242752 bytes 2026-02-20T22:43:16.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4599800832 bytes 2026-02-20T22:43:16.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4660584448 bytes 2026-02-20T22:43:16.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4690028544 bytes 2026-02-20T22:43:16.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4690660352 bytes 2026-02-20T22:43:16.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4738044928 bytes 2026-02-20T22:43:17.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4794786816 bytes 2026-02-20T22:43:17.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4823876608 bytes 2026-02-20T22:43:17.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4846465024 bytes 2026-02-20T22:43:17.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4900074496 bytes 2026-02-20T22:43:17.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4926039040 bytes 2026-02-20T22:43:17.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4939162624 bytes 2026-02-20T22:43:17.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4994154496 bytes 2026-02-20T22:43:17.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5034736640 bytes 2026-02-20T22:43:17.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5083851776 bytes 2026-02-20T22:43:17.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5110950912 bytes 2026-02-20T22:43:17.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5111577600 bytes 2026-02-20T22:43:17.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5196732416 bytes 2026-02-20T22:43:17.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5206689792 bytes 2026-02-20T22:43:17.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5207282688 bytes 2026-02-20T22:43:17.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5307196416 bytes 2026-02-20T22:43:17.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5413354496 bytes 2026-02-20T22:43:17.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5482858496 bytes 2026-02-20T22:43:17.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5534148608 bytes 2026-02-20T22:43:18.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5557156864 bytes 2026-02-20T22:43:18.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5562763264 bytes 2026-02-20T22:43:18.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5578238976 bytes 2026-02-20T22:43:18.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5660029952 bytes 2026-02-20T22:43:18.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5753255936 bytes 2026-02-20T22:43:18.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5783164928 bytes 2026-02-20T22:43:18.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5803556864 bytes 2026-02-20T22:43:18.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5813639168 bytes 2026-02-20T22:43:18.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5866420224 bytes 2026-02-20T22:43:18.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5942250496 bytes 2026-02-20T22:43:18.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5970376704 bytes 2026-02-20T22:43:18.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5982749696 bytes 2026-02-20T22:43:18.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6016607232 bytes 2026-02-20T22:43:18.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6098560000 bytes 2026-02-20T22:43:19.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6195466240 bytes 2026-02-20T22:43:19.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6299502592 bytes 2026-02-20T22:43:19.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6316237824 bytes 2026-02-20T22:43:19.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6358619136 bytes 2026-02-20T22:43:19.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6393879552 bytes 2026-02-20T22:43:19.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6395201536 bytes 2026-02-20T22:43:19.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6429395968 bytes 2026-02-20T22:43:19.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6476301312 bytes 2026-02-20T22:43:19.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6527270912 bytes 2026-02-20T22:43:19.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6528834560 bytes 2026-02-20T22:43:19.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6545445888 bytes 2026-02-20T22:43:19.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6595621888 bytes 2026-02-20T22:43:19.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6678995968 bytes 2026-02-20T22:43:19.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6720300032 bytes 2026-02-20T22:43:19.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6786782208 bytes 2026-02-20T22:43:19.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6823172096 bytes 2026-02-20T22:43:19.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6849489920 bytes 2026-02-20T22:43:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6880488448 bytes 2026-02-20T22:43:20.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6899627008 bytes 2026-02-20T22:43:20.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6924701696 bytes 2026-02-20T22:43:20.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7006391296 bytes 2026-02-20T22:43:20.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7026611200 bytes 2026-02-20T22:43:20.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7095959552 bytes 2026-02-20T22:43:20.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7117918208 bytes 2026-02-20T22:43:20.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7171601408 bytes 2026-02-20T22:43:20.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7228557312 bytes 2026-02-20T22:43:20.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7280381952 bytes 2026-02-20T22:43:20.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7301756928 bytes 2026-02-20T22:43:20.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7339924480 bytes 2026-02-20T22:43:20.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7353868288 bytes 2026-02-20T22:43:20.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7368973312 bytes 2026-02-20T22:43:20.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7433417728 bytes 2026-02-20T22:43:20.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7476485120 bytes 2026-02-20T22:43:20.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7477122048 bytes 2026-02-20T22:43:20.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7533106176 bytes 2026-02-20T22:43:20.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7585507328 bytes 2026-02-20T22:43:21.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7621690368 bytes 2026-02-20T22:43:21.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7625189376 bytes 2026-02-20T22:43:21.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7711827968 bytes 2026-02-20T22:43:21.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7768644608 bytes 2026-02-20T22:43:21.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7776305152 bytes 2026-02-20T22:43:21.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7797861376 bytes 2026-02-20T22:43:21.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7837084672 bytes 2026-02-20T22:43:21.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7876713472 bytes 2026-02-20T22:43:21.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7947564032 bytes 2026-02-20T22:43:21.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7978844160 bytes 2026-02-20T22:43:21.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7987953664 bytes 2026-02-20T22:43:21.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8022074368 bytes 2026-02-20T22:43:21.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8073840640 bytes 2026-02-20T22:43:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8077843456 bytes 2026-02-20T22:43:21.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8182489088 bytes 2026-02-20T22:43:21.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8258106368 bytes 2026-02-20T22:43:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8311771136 bytes 2026-02-20T22:43:21.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8332088320 bytes 2026-02-20T22:43:22.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8382056448 bytes 2026-02-20T22:43:22.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8426440704 bytes 2026-02-20T22:43:22.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8560524288 bytes 2026-02-20T22:43:22.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8623618048 bytes 2026-02-20T22:43:22.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8637368320 bytes 2026-02-20T22:43:22.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8670936064 bytes 2026-02-20T22:43:22.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8709248000 bytes 2026-02-20T22:43:22.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8712807424 bytes 2026-02-20T22:43:22.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8746071040 bytes 2026-02-20T22:43:22.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8770605056 bytes 2026-02-20T22:43:22.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8845292544 bytes 2026-02-20T22:43:22.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8889358336 bytes 2026-02-20T22:43:22.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8909768704 bytes 2026-02-20T22:43:22.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8950402048 bytes 2026-02-20T22:43:22.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8998378496 bytes 2026-02-20T22:43:22.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9021496320 bytes 2026-02-20T22:43:22.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9102905344 bytes 2026-02-20T22:43:23.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9111654400 bytes 2026-02-20T22:43:23.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9195706368 bytes 2026-02-20T22:43:23.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9303158784 bytes 2026-02-20T22:43:23.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9341808640 bytes 2026-02-20T22:43:23.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9394121728 bytes 2026-02-20T22:43:23.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9431200768 bytes 2026-02-20T22:43:23.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9436255232 bytes 2026-02-20T22:43:23.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9489782784 bytes 2026-02-20T22:43:23.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9515478016 bytes 2026-02-20T22:43:23.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9556169728 bytes 2026-02-20T22:43:23.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9571060736 bytes 2026-02-20T22:43:23.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9601654784 bytes 2026-02-20T22:43:23.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9669237760 bytes 2026-02-20T22:43:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9732536320 bytes 2026-02-20T22:43:23.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9758569472 bytes 2026-02-20T22:43:23.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9800338432 bytes 2026-02-20T22:43:23.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 9833688064 bytes 2026-02-20T22:43:24.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 9943422976 bytes 2026-02-20T22:43:24.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 9957625856 bytes 2026-02-20T22:43:24.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 9989610496 bytes 2026-02-20T22:43:24.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 10014754816 bytes 2026-02-20T22:43:24.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 10085156864 bytes 2026-02-20T22:43:24.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23200 IOs, 10122061824 bytes 2026-02-20T22:43:24.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23300 IOs, 10179123200 bytes 2026-02-20T22:43:24.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23400 IOs, 10202150912 bytes 2026-02-20T22:43:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23500 IOs, 10236532736 bytes 2026-02-20T22:43:24.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23600 IOs, 10280681472 bytes 2026-02-20T22:43:24.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23700 IOs, 10351093760 bytes 2026-02-20T22:43:24.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23800 IOs, 10380819456 bytes 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_45 2026-02-20T22:43:24.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_45 128M 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2026-02-20T22:43:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:43:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:24.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2026-02-20T22:43:25.073 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:43:25.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:43:25.073 INFO:tasks.rbd_mirror_thrash:waiting for 2.64991173081792 secs before reviving daemons 2026-02-20T22:43:25.074 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:43:25.069+0000 7f7206226640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 30358) UID: 1000 2026-02-20T22:43:25.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:25.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_45': 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 14522ef2274 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.14522ef2274 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:24 2026 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:24 2026 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:24 2026 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ebe686cb-a3fc-48a1-a990-521e037b144f 2026-02-20T22:43:25.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_45 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:43:25.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:43:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:25.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:25.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 50972672 bytes 2026-02-20T22:43:25.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 51589120 bytes 2026-02-20T22:43:25.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 145802240 bytes 2026-02-20T22:43:26.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 192778240 bytes 2026-02-20T22:43:26.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 243301376 bytes 2026-02-20T22:43:26.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 303376384 bytes 2026-02-20T22:43:26.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 345131008 bytes 2026-02-20T22:43:26.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 377627648 bytes 2026-02-20T22:43:26.294 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:26.285+0000 7f7ed1951640 -1 reset not still connected to 0x55c72112add0 2026-02-20T22:43:26.294 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:26.285+0000 7f7ed1951640 -1 reset not still connected to 0x55c722c8e8f0 2026-02-20T22:43:26.295 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:26.285+0000 7f7ed1951640 -1 reset not still connected to 0x55c73ef8bd40 2026-02-20T22:43:26.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 439219200 bytes 2026-02-20T22:43:26.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 468268032 bytes 2026-02-20T22:43:26.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 487291904 bytes 2026-02-20T22:43:26.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 509782016 bytes 2026-02-20T22:43:26.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 579799040 bytes 2026-02-20T22:43:26.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 593970176 bytes 2026-02-20T22:43:26.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 626463744 bytes 2026-02-20T22:43:26.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 637263872 bytes 2026-02-20T22:43:26.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 685995008 bytes 2026-02-20T22:43:26.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 719632384 bytes 2026-02-20T22:43:26.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 743397376 bytes 2026-02-20T22:43:26.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 760401920 bytes 2026-02-20T22:43:26.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 794109952 bytes 2026-02-20T22:43:26.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 815750144 bytes 2026-02-20T22:43:27.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 915581952 bytes 2026-02-20T22:43:27.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 940258304 bytes 2026-02-20T22:43:27.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 968101888 bytes 2026-02-20T22:43:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1035907072 bytes 2026-02-20T22:43:27.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1057629184 bytes 2026-02-20T22:43:27.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1115827200 bytes 2026-02-20T22:43:27.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1158433792 bytes 2026-02-20T22:43:27.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1201532928 bytes 2026-02-20T22:43:27.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1228959744 bytes 2026-02-20T22:43:27.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1242381312 bytes 2026-02-20T22:43:27.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1286621184 bytes 2026-02-20T22:43:27.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1327623168 bytes 2026-02-20T22:43:27.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1362459648 bytes 2026-02-20T22:43:27.724 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:43:27.724 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:43:27.725 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:43:27.725 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:43:27.725 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:43:27.727 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:43:27.727 INFO:tasks.rbd_mirror_thrash:waiting for 69.97351663828604 secs before thrashing 2026-02-20T22:43:27.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1449742336 bytes 2026-02-20T22:43:28.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1480212480 bytes 2026-02-20T22:43:28.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1485887488 bytes 2026-02-20T22:43:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1506947072 bytes 2026-02-20T22:43:28.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1536603136 bytes 2026-02-20T22:43:28.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1567495168 bytes 2026-02-20T22:43:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1600376832 bytes 2026-02-20T22:43:28.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1648984064 bytes 2026-02-20T22:43:28.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1661753344 bytes 2026-02-20T22:43:28.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1690435584 bytes 2026-02-20T22:43:28.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1702749184 bytes 2026-02-20T22:43:28.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1762751488 bytes 2026-02-20T22:43:28.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1823052800 bytes 2026-02-20T22:43:28.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1851094016 bytes 2026-02-20T22:43:28.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1895081984 bytes 2026-02-20T22:43:28.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1947475968 bytes 2026-02-20T22:43:28.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1974815744 bytes 2026-02-20T22:43:28.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2017315840 bytes 2026-02-20T22:43:28.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2102211584 bytes 2026-02-20T22:43:29.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2155019264 bytes 2026-02-20T22:43:29.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2169373696 bytes 2026-02-20T22:43:29.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2209135616 bytes 2026-02-20T22:43:29.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2241817600 bytes 2026-02-20T22:43:29.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2264414208 bytes 2026-02-20T22:43:29.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2287067136 bytes 2026-02-20T22:43:29.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2328971264 bytes 2026-02-20T22:43:29.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2363535360 bytes 2026-02-20T22:43:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2461366272 bytes 2026-02-20T22:43:29.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2566605824 bytes 2026-02-20T22:43:29.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2615879680 bytes 2026-02-20T22:43:29.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2675740672 bytes 2026-02-20T22:43:29.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2729612288 bytes 2026-02-20T22:43:29.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2755833856 bytes 2026-02-20T22:43:29.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2784892928 bytes 2026-02-20T22:43:29.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2791213056 bytes 2026-02-20T22:43:29.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2831825920 bytes 2026-02-20T22:43:30.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2898491392 bytes 2026-02-20T22:43:30.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2922635264 bytes 2026-02-20T22:43:30.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3017791488 bytes 2026-02-20T22:43:30.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3102674944 bytes 2026-02-20T22:43:30.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3118721024 bytes 2026-02-20T22:43:30.405 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3141771264 bytes 2026-02-20T22:43:30.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3184809984 bytes 2026-02-20T22:43:30.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3185426432 bytes 2026-02-20T22:43:30.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3222183936 bytes 2026-02-20T22:43:30.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3278410752 bytes 2026-02-20T22:43:30.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3365170176 bytes 2026-02-20T22:43:30.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3385056256 bytes 2026-02-20T22:43:30.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3417621504 bytes 2026-02-20T22:43:30.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3477591040 bytes 2026-02-20T22:43:30.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3528204288 bytes 2026-02-20T22:43:30.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3529683968 bytes 2026-02-20T22:43:31.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3570232320 bytes 2026-02-20T22:43:31.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3605253120 bytes 2026-02-20T22:43:31.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3636986880 bytes 2026-02-20T22:43:31.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3708050432 bytes 2026-02-20T22:43:31.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3722535936 bytes 2026-02-20T22:43:31.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3767154688 bytes 2026-02-20T22:43:31.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3837582336 bytes 2026-02-20T22:43:31.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3878566912 bytes 2026-02-20T22:43:31.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3896869888 bytes 2026-02-20T22:43:31.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3937523712 bytes 2026-02-20T22:43:31.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3964484608 bytes 2026-02-20T22:43:31.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4007586816 bytes 2026-02-20T22:43:31.656 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:31.649+0000 7f7ed1951640 -1 reset not still connected to 0x55c70bf0c5b0 2026-02-20T22:43:31.656 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:31.649+0000 7f7ed1951640 -1 reset not still connected to 0x55c71ce92680 2026-02-20T22:43:31.657 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:43:31.649+0000 7f7ed1951640 -1 reset not still connected to 0x55c73ef8b380 2026-02-20T22:43:31.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4035080192 bytes 2026-02-20T22:43:31.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4064686080 bytes 2026-02-20T22:43:31.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4144218112 bytes 2026-02-20T22:43:31.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4218875904 bytes 2026-02-20T22:43:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4249311232 bytes 2026-02-20T22:43:32.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4326953984 bytes 2026-02-20T22:43:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4348489728 bytes 2026-02-20T22:43:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4399782912 bytes 2026-02-20T22:43:32.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4446245888 bytes 2026-02-20T22:43:32.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4481420288 bytes 2026-02-20T22:43:32.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4486943744 bytes 2026-02-20T22:43:32.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4524242944 bytes 2026-02-20T22:43:32.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4524864512 bytes 2026-02-20T22:43:32.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4552337408 bytes 2026-02-20T22:43:32.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4586108928 bytes 2026-02-20T22:43:32.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4653286400 bytes 2026-02-20T22:43:32.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4653886464 bytes 2026-02-20T22:43:32.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4728571904 bytes 2026-02-20T22:43:32.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4743590912 bytes 2026-02-20T22:43:32.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4796503040 bytes 2026-02-20T22:43:32.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4820121600 bytes 2026-02-20T22:43:32.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4852897792 bytes 2026-02-20T22:43:32.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4901256192 bytes 2026-02-20T22:43:32.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4959233024 bytes 2026-02-20T22:43:33.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5008594944 bytes 2026-02-20T22:43:33.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5038410752 bytes 2026-02-20T22:43:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5104522240 bytes 2026-02-20T22:43:33.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5162928128 bytes 2026-02-20T22:43:33.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5180824576 bytes 2026-02-20T22:43:33.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5206839296 bytes 2026-02-20T22:43:33.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5244830720 bytes 2026-02-20T22:43:33.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5280732160 bytes 2026-02-20T22:43:33.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5343905792 bytes 2026-02-20T22:43:33.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5395421184 bytes 2026-02-20T22:43:33.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5416635392 bytes 2026-02-20T22:43:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5442035712 bytes 2026-02-20T22:43:33.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5496825856 bytes 2026-02-20T22:43:33.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5542875136 bytes 2026-02-20T22:43:33.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5565070336 bytes 2026-02-20T22:43:33.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5633201152 bytes 2026-02-20T22:43:33.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5657277440 bytes 2026-02-20T22:43:33.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5681685504 bytes 2026-02-20T22:43:34.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5712072704 bytes 2026-02-20T22:43:34.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5747499008 bytes 2026-02-20T22:43:34.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5807305728 bytes 2026-02-20T22:43:34.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5897662464 bytes 2026-02-20T22:43:34.335 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 5913116672 bytes 2026-02-20T22:43:34.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 5966647296 bytes 2026-02-20T22:43:34.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 5994578944 bytes 2026-02-20T22:43:34.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6031166464 bytes 2026-02-20T22:43:34.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6068596736 bytes 2026-02-20T22:43:34.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6128237568 bytes 2026-02-20T22:43:34.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6193116160 bytes 2026-02-20T22:43:34.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6193736704 bytes 2026-02-20T22:43:34.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6247643136 bytes 2026-02-20T22:43:34.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6304532480 bytes 2026-02-20T22:43:34.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6360124416 bytes 2026-02-20T22:43:35.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6421626880 bytes 2026-02-20T22:43:35.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6447207424 bytes 2026-02-20T22:43:35.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6464772096 bytes 2026-02-20T22:43:35.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6522936320 bytes 2026-02-20T22:43:35.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6549829632 bytes 2026-02-20T22:43:35.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6639529984 bytes 2026-02-20T22:43:35.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6671379456 bytes 2026-02-20T22:43:35.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6739667968 bytes 2026-02-20T22:43:35.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6771226624 bytes 2026-02-20T22:43:35.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6774490112 bytes 2026-02-20T22:43:35.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 6789419008 bytes 2026-02-20T22:43:35.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 6803386368 bytes 2026-02-20T22:43:35.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 6849910784 bytes 2026-02-20T22:43:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 6894747648 bytes 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:43:35.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_46 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_46 128M 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:35.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:43:35.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:35.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2026-02-20T22:43:36.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:36.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2026-02-20T22:43:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_46': 2026-02-20T22:43:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:36.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 146588d5a40 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.146588d5a40 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:35 2026 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:35 2026 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:35 2026 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 0f4136cd-d44e-415e-a02d-e8635f545728 2026-02-20T22:43:36.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_46 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:43:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=8 2026-02-20T22:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:36.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:36.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 22559744 bytes 2026-02-20T22:43:36.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 93834240 bytes 2026-02-20T22:43:37.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 155530240 bytes 2026-02-20T22:43:37.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 222261248 bytes 2026-02-20T22:43:37.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 259968000 bytes 2026-02-20T22:43:37.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 318831616 bytes 2026-02-20T22:43:37.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 407784448 bytes 2026-02-20T22:43:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 423312384 bytes 2026-02-20T22:43:37.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 462097408 bytes 2026-02-20T22:43:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 464614400 bytes 2026-02-20T22:43:37.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 488238080 bytes 2026-02-20T22:43:37.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 514556928 bytes 2026-02-20T22:43:37.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 557820928 bytes 2026-02-20T22:43:38.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 604988416 bytes 2026-02-20T22:43:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 622289920 bytes 2026-02-20T22:43:38.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 727857152 bytes 2026-02-20T22:43:38.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 773239808 bytes 2026-02-20T22:43:38.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 842433536 bytes 2026-02-20T22:43:38.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 922812416 bytes 2026-02-20T22:43:38.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 966073344 bytes 2026-02-20T22:43:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 977923072 bytes 2026-02-20T22:43:38.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 997369856 bytes 2026-02-20T22:43:38.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 999176192 bytes 2026-02-20T22:43:38.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1058370560 bytes 2026-02-20T22:43:38.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1092580352 bytes 2026-02-20T22:43:38.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1142605824 bytes 2026-02-20T22:43:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1186614272 bytes 2026-02-20T22:43:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1244809216 bytes 2026-02-20T22:43:39.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1279885312 bytes 2026-02-20T22:43:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1347103744 bytes 2026-02-20T22:43:39.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1357491200 bytes 2026-02-20T22:43:39.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1377496064 bytes 2026-02-20T22:43:39.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1417056256 bytes 2026-02-20T22:43:39.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1439905792 bytes 2026-02-20T22:43:39.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1476344832 bytes 2026-02-20T22:43:39.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1507449856 bytes 2026-02-20T22:43:39.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1537236992 bytes 2026-02-20T22:43:39.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1556722688 bytes 2026-02-20T22:43:39.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1601583104 bytes 2026-02-20T22:43:39.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1661426688 bytes 2026-02-20T22:43:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1694915584 bytes 2026-02-20T22:43:39.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1781134336 bytes 2026-02-20T22:43:39.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1799688192 bytes 2026-02-20T22:43:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1820540928 bytes 2026-02-20T22:43:40.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1933940736 bytes 2026-02-20T22:43:40.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1949655040 bytes 2026-02-20T22:43:40.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1964839936 bytes 2026-02-20T22:43:40.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1989634048 bytes 2026-02-20T22:43:40.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1990261760 bytes 2026-02-20T22:43:40.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2038337536 bytes 2026-02-20T22:43:40.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2069102592 bytes 2026-02-20T22:43:40.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2078870528 bytes 2026-02-20T22:43:40.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2138816512 bytes 2026-02-20T22:43:40.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2151829504 bytes 2026-02-20T22:43:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2176732160 bytes 2026-02-20T22:43:40.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2185176064 bytes 2026-02-20T22:43:40.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2205665280 bytes 2026-02-20T22:43:40.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2261386240 bytes 2026-02-20T22:43:40.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2321299456 bytes 2026-02-20T22:43:40.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2349548544 bytes 2026-02-20T22:43:40.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2364910592 bytes 2026-02-20T22:43:40.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2452851712 bytes 2026-02-20T22:43:41.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2518168576 bytes 2026-02-20T22:43:41.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2528986112 bytes 2026-02-20T22:43:41.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2546561024 bytes 2026-02-20T22:43:41.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2643686400 bytes 2026-02-20T22:43:41.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2700391424 bytes 2026-02-20T22:43:41.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2721813504 bytes 2026-02-20T22:43:41.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2767119360 bytes 2026-02-20T22:43:41.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2822863872 bytes 2026-02-20T22:43:41.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2901482496 bytes 2026-02-20T22:43:41.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2934635520 bytes 2026-02-20T22:43:41.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2965315584 bytes 2026-02-20T22:43:41.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2990784512 bytes 2026-02-20T22:43:41.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2991409152 bytes 2026-02-20T22:43:41.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3045352448 bytes 2026-02-20T22:43:41.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3077075968 bytes 2026-02-20T22:43:41.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3142335488 bytes 2026-02-20T22:43:41.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3189281792 bytes 2026-02-20T22:43:42.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3226847232 bytes 2026-02-20T22:43:42.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3273345024 bytes 2026-02-20T22:43:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3379614720 bytes 2026-02-20T22:43:42.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3434814464 bytes 2026-02-20T22:43:42.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3436301312 bytes 2026-02-20T22:43:42.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3451555840 bytes 2026-02-20T22:43:42.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3522591744 bytes 2026-02-20T22:43:42.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3542590464 bytes 2026-02-20T22:43:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3586258944 bytes 2026-02-20T22:43:42.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3605129216 bytes 2026-02-20T22:43:42.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3662247936 bytes 2026-02-20T22:43:42.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3719112704 bytes 2026-02-20T22:43:42.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3782805504 bytes 2026-02-20T22:43:42.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3821727744 bytes 2026-02-20T22:43:42.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3852336128 bytes 2026-02-20T22:43:42.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3979091968 bytes 2026-02-20T22:43:42.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4007837696 bytes 2026-02-20T22:43:43.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4069476352 bytes 2026-02-20T22:43:43.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4103899136 bytes 2026-02-20T22:43:43.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4125481984 bytes 2026-02-20T22:43:43.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4208595968 bytes 2026-02-20T22:43:43.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4242502656 bytes 2026-02-20T22:43:43.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4298204160 bytes 2026-02-20T22:43:43.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4351447040 bytes 2026-02-20T22:43:43.468 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4363423744 bytes 2026-02-20T22:43:43.544 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4425441280 bytes 2026-02-20T22:43:43.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4463461376 bytes 2026-02-20T22:43:43.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4493257728 bytes 2026-02-20T22:43:43.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4563571712 bytes 2026-02-20T22:43:43.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4600395776 bytes 2026-02-20T22:43:43.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4671293440 bytes 2026-02-20T22:43:43.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4687416320 bytes 2026-02-20T22:43:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4723132416 bytes 2026-02-20T22:43:44.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4772788224 bytes 2026-02-20T22:43:44.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4859216896 bytes 2026-02-20T22:43:44.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4882263040 bytes 2026-02-20T22:43:44.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4925808640 bytes 2026-02-20T22:43:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4950197248 bytes 2026-02-20T22:43:44.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5017315328 bytes 2026-02-20T22:43:44.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5097245696 bytes 2026-02-20T22:43:44.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5154324480 bytes 2026-02-20T22:43:44.535 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5222865920 bytes 2026-02-20T22:43:44.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5249453056 bytes 2026-02-20T22:43:44.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5285519360 bytes 2026-02-20T22:43:44.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5360830464 bytes 2026-02-20T22:43:44.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5393785856 bytes 2026-02-20T22:43:44.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_47 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_47 128M 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:44.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2026-02-20T22:43:44.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:44.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2026-02-20T22:43:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:45.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_47': 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 146d337b8d4f 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.146d337b8d4f 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:45.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:44 2026 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:44 2026 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:44 2026 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: a7ff2d10-0af9-4bfa-8665-8b032dd2395c 2026-02-20T22:43:45.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_47 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:43:45.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:43:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:45.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:45.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 43409408 bytes 2026-02-20T22:43:45.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 93498368 bytes 2026-02-20T22:43:46.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 170766336 bytes 2026-02-20T22:43:46.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 217678848 bytes 2026-02-20T22:43:46.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 242390016 bytes 2026-02-20T22:43:46.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 303930368 bytes 2026-02-20T22:43:46.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 378703872 bytes 2026-02-20T22:43:46.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 458457088 bytes 2026-02-20T22:43:46.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 512589824 bytes 2026-02-20T22:43:46.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 537976832 bytes 2026-02-20T22:43:46.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 569022464 bytes 2026-02-20T22:43:46.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 640345088 bytes 2026-02-20T22:43:46.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 662605824 bytes 2026-02-20T22:43:46.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 677322752 bytes 2026-02-20T22:43:46.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 761252864 bytes 2026-02-20T22:43:46.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 784212992 bytes 2026-02-20T22:43:46.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 845942784 bytes 2026-02-20T22:43:46.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 880151552 bytes 2026-02-20T22:43:46.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 925634560 bytes 2026-02-20T22:43:47.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 972324864 bytes 2026-02-20T22:43:47.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1005650944 bytes 2026-02-20T22:43:47.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1065523200 bytes 2026-02-20T22:43:47.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1101066240 bytes 2026-02-20T22:43:47.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1204749312 bytes 2026-02-20T22:43:47.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1215968256 bytes 2026-02-20T22:43:47.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1233410048 bytes 2026-02-20T22:43:47.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1314172928 bytes 2026-02-20T22:43:47.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1318138880 bytes 2026-02-20T22:43:47.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1430572032 bytes 2026-02-20T22:43:47.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1451483136 bytes 2026-02-20T22:43:47.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1486599168 bytes 2026-02-20T22:43:47.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1551013888 bytes 2026-02-20T22:43:47.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1570975744 bytes 2026-02-20T22:43:47.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1571580928 bytes 2026-02-20T22:43:47.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1609181184 bytes 2026-02-20T22:43:47.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1640604672 bytes 2026-02-20T22:43:47.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1658469376 bytes 2026-02-20T22:43:48.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1686585344 bytes 2026-02-20T22:43:48.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1722301440 bytes 2026-02-20T22:43:48.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1765852160 bytes 2026-02-20T22:43:48.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1823185920 bytes 2026-02-20T22:43:48.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1868556288 bytes 2026-02-20T22:43:48.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1906659328 bytes 2026-02-20T22:43:48.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2003602432 bytes 2026-02-20T22:43:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2004212736 bytes 2026-02-20T22:43:48.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2100460544 bytes 2026-02-20T22:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2121113600 bytes 2026-02-20T22:43:48.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2163228672 bytes 2026-02-20T22:43:48.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2214003712 bytes 2026-02-20T22:43:48.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2257580032 bytes 2026-02-20T22:43:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2304560128 bytes 2026-02-20T22:43:48.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2372255744 bytes 2026-02-20T22:43:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2404978688 bytes 2026-02-20T22:43:49.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2434975744 bytes 2026-02-20T22:43:49.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2478359552 bytes 2026-02-20T22:43:49.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2513189888 bytes 2026-02-20T22:43:49.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2521540608 bytes 2026-02-20T22:43:49.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2613972992 bytes 2026-02-20T22:43:49.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2647045120 bytes 2026-02-20T22:43:49.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2702500864 bytes 2026-02-20T22:43:49.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2746451968 bytes 2026-02-20T22:43:49.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2799295488 bytes 2026-02-20T22:43:49.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2833291264 bytes 2026-02-20T22:43:49.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2891701248 bytes 2026-02-20T22:43:49.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2951285760 bytes 2026-02-20T22:43:49.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3026173952 bytes 2026-02-20T22:43:49.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3091097600 bytes 2026-02-20T22:43:49.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3137650688 bytes 2026-02-20T22:43:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3220133888 bytes 2026-02-20T22:43:50.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3257101312 bytes 2026-02-20T22:43:50.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3314046976 bytes 2026-02-20T22:43:50.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3335915520 bytes 2026-02-20T22:43:50.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3353502720 bytes 2026-02-20T22:43:50.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3377158144 bytes 2026-02-20T22:43:50.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3451928576 bytes 2026-02-20T22:43:50.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3483749376 bytes 2026-02-20T22:43:50.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3493460992 bytes 2026-02-20T22:43:50.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3539200000 bytes 2026-02-20T22:43:50.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3539792896 bytes 2026-02-20T22:43:50.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3560668160 bytes 2026-02-20T22:43:50.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3695839232 bytes 2026-02-20T22:43:50.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3766827008 bytes 2026-02-20T22:43:50.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3812156416 bytes 2026-02-20T22:43:50.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3828569088 bytes 2026-02-20T22:43:50.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3854264320 bytes 2026-02-20T22:43:50.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3913917440 bytes 2026-02-20T22:43:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3982280704 bytes 2026-02-20T22:43:51.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4040441856 bytes 2026-02-20T22:43:51.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4093336576 bytes 2026-02-20T22:43:51.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4128695296 bytes 2026-02-20T22:43:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4161209344 bytes 2026-02-20T22:43:51.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4198510592 bytes 2026-02-20T22:43:51.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4246276096 bytes 2026-02-20T22:43:51.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4264310784 bytes 2026-02-20T22:43:51.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4276664320 bytes 2026-02-20T22:43:51.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4321466368 bytes 2026-02-20T22:43:51.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4345761792 bytes 2026-02-20T22:43:51.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4407713792 bytes 2026-02-20T22:43:51.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4462340096 bytes 2026-02-20T22:43:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4563244032 bytes 2026-02-20T22:43:51.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4563837952 bytes 2026-02-20T22:43:51.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4605991936 bytes 2026-02-20T22:43:51.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4617493504 bytes 2026-02-20T22:43:52.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4676100096 bytes 2026-02-20T22:43:52.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4690991104 bytes 2026-02-20T22:43:52.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4694890496 bytes 2026-02-20T22:43:52.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4758895616 bytes 2026-02-20T22:43:52.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4772927488 bytes 2026-02-20T22:43:52.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4829264896 bytes 2026-02-20T22:43:52.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4863043584 bytes 2026-02-20T22:43:52.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4879973376 bytes 2026-02-20T22:43:52.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4917677056 bytes 2026-02-20T22:43:52.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4927361024 bytes 2026-02-20T22:43:52.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4991637504 bytes 2026-02-20T22:43:52.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5022564352 bytes 2026-02-20T22:43:52.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5114915840 bytes 2026-02-20T22:43:52.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5169018880 bytes 2026-02-20T22:43:52.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5211764736 bytes 2026-02-20T22:43:52.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5267920896 bytes 2026-02-20T22:43:52.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5315951616 bytes 2026-02-20T22:43:52.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5350473728 bytes 2026-02-20T22:43:52.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5367559168 bytes 2026-02-20T22:43:53.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5382449152 bytes 2026-02-20T22:43:53.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5430768640 bytes 2026-02-20T22:43:53.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5477744640 bytes 2026-02-20T22:43:53.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5523538944 bytes 2026-02-20T22:43:53.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5562189824 bytes 2026-02-20T22:43:53.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5622103040 bytes 2026-02-20T22:43:53.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5661365248 bytes 2026-02-20T22:43:53.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5710536704 bytes 2026-02-20T22:43:53.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5765718016 bytes 2026-02-20T22:43:53.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5824030720 bytes 2026-02-20T22:43:53.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5881816064 bytes 2026-02-20T22:43:53.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5911818240 bytes 2026-02-20T22:43:53.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5979258880 bytes 2026-02-20T22:43:53.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6046649344 bytes 2026-02-20T22:43:53.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6079498240 bytes 2026-02-20T22:43:53.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6148462592 bytes 2026-02-20T22:43:54.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6168673280 bytes 2026-02-20T22:43:54.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6205068288 bytes 2026-02-20T22:43:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6301054976 bytes 2026-02-20T22:43:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6346742784 bytes 2026-02-20T22:43:54.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6365739008 bytes 2026-02-20T22:43:54.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6368270336 bytes 2026-02-20T22:43:54.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6459123712 bytes 2026-02-20T22:43:54.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6559932416 bytes 2026-02-20T22:43:54.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6650353664 bytes 2026-02-20T22:43:54.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6656878592 bytes 2026-02-20T22:43:54.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6692372480 bytes 2026-02-20T22:43:54.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6721050624 bytes 2026-02-20T22:43:54.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6758075392 bytes 2026-02-20T22:43:54.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6789169152 bytes 2026-02-20T22:43:55.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6822795264 bytes 2026-02-20T22:43:55.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6834934784 bytes 2026-02-20T22:43:55.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6838923264 bytes 2026-02-20T22:43:55.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6880604160 bytes 2026-02-20T22:43:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6908036096 bytes 2026-02-20T22:43:55.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6977646592 bytes 2026-02-20T22:43:55.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6994527232 bytes 2026-02-20T22:43:55.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7073839104 bytes 2026-02-20T22:43:55.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7091757056 bytes 2026-02-20T22:43:55.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7110152192 bytes 2026-02-20T22:43:55.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7229189120 bytes 2026-02-20T22:43:55.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7259816960 bytes 2026-02-20T22:43:55.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7304402944 bytes 2026-02-20T22:43:55.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_48 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:55.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_48 128M 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:43:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:43:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:43:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:43:55.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2026-02-20T22:43:55.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:43:55.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:43:55.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2026-02-20T22:43:56.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:43:56.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2026-02-20T22:43:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_48': 2026-02-20T22:43:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:43:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:43:56.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 147692f01da9 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.147692f01da9 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:55 2026 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:55 2026 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:55 2026 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:43:56.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 8397c19c-505f-4fea-8ee6-6d3f6b4a2bbd 2026-02-20T22:43:56.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_48 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:43:56.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:43:56.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:43:56.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:43:56.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:43:56.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 38042624 bytes 2026-02-20T22:43:56.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 77301760 bytes 2026-02-20T22:43:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 77895680 bytes 2026-02-20T22:43:57.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 122517504 bytes 2026-02-20T22:43:57.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 173874176 bytes 2026-02-20T22:43:57.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 212236288 bytes 2026-02-20T22:43:57.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 264869888 bytes 2026-02-20T22:43:57.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 325653504 bytes 2026-02-20T22:43:57.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 399405056 bytes 2026-02-20T22:43:57.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 414541824 bytes 2026-02-20T22:43:57.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 425552896 bytes 2026-02-20T22:43:57.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 462779392 bytes 2026-02-20T22:43:57.557 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 502403072 bytes 2026-02-20T22:43:57.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 518139904 bytes 2026-02-20T22:43:57.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 554716160 bytes 2026-02-20T22:43:57.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 566351872 bytes 2026-02-20T22:43:57.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 593840128 bytes 2026-02-20T22:43:57.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 660458496 bytes 2026-02-20T22:43:57.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 708939776 bytes 2026-02-20T22:43:57.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 805412864 bytes 2026-02-20T22:43:57.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 822663168 bytes 2026-02-20T22:43:58.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 863462400 bytes 2026-02-20T22:43:58.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 973946880 bytes 2026-02-20T22:43:58.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 999516160 bytes 2026-02-20T22:43:58.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1009445888 bytes 2026-02-20T22:43:58.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1019429888 bytes 2026-02-20T22:43:58.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1061001216 bytes 2026-02-20T22:43:58.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1132312576 bytes 2026-02-20T22:43:58.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1160867840 bytes 2026-02-20T22:43:58.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1232873472 bytes 2026-02-20T22:43:58.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1324272640 bytes 2026-02-20T22:43:58.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1342941184 bytes 2026-02-20T22:43:58.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1415350272 bytes 2026-02-20T22:43:58.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1479834624 bytes 2026-02-20T22:43:58.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1498218496 bytes 2026-02-20T22:43:58.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1552967680 bytes 2026-02-20T22:43:58.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1595038720 bytes 2026-02-20T22:43:59.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1607646208 bytes 2026-02-20T22:43:59.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1642468352 bytes 2026-02-20T22:43:59.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1643077632 bytes 2026-02-20T22:43:59.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1683584000 bytes 2026-02-20T22:43:59.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1684208640 bytes 2026-02-20T22:43:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1769780224 bytes 2026-02-20T22:43:59.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1810951168 bytes 2026-02-20T22:43:59.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1811529728 bytes 2026-02-20T22:43:59.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1846306816 bytes 2026-02-20T22:43:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1905569792 bytes 2026-02-20T22:43:59.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1988820992 bytes 2026-02-20T22:43:59.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2025558016 bytes 2026-02-20T22:43:59.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2080889856 bytes 2026-02-20T22:43:59.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2107036672 bytes 2026-02-20T22:43:59.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2177338368 bytes 2026-02-20T22:43:59.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2188516352 bytes 2026-02-20T22:43:59.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2205205504 bytes 2026-02-20T22:43:59.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2207180800 bytes 2026-02-20T22:43:59.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2234615808 bytes 2026-02-20T22:44:00.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2235219968 bytes 2026-02-20T22:44:00.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2249672704 bytes 2026-02-20T22:44:00.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7880452367682872) > weight (0.25) 2026-02-20T22:44:00.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9647950592054036) > weight (0.25) 2026-02-20T22:44:00.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.42913934810991095) > weight (0.25) 2026-02-20T22:44:00.075 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6486392981476813) > weight (0.25) 2026-02-20T22:44:00.076 INFO:tasks.rbd_mirror_thrash:waiting for 99.89776528281767 secs before thrashing 2026-02-20T22:44:00.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2335166464 bytes 2026-02-20T22:44:00.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2442690560 bytes 2026-02-20T22:44:00.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2472115200 bytes 2026-02-20T22:44:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2493880320 bytes 2026-02-20T22:44:00.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2516037632 bytes 2026-02-20T22:44:00.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2562882560 bytes 2026-02-20T22:44:00.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2626489344 bytes 2026-02-20T22:44:00.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2640023552 bytes 2026-02-20T22:44:00.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2713835520 bytes 2026-02-20T22:44:00.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2737525760 bytes 2026-02-20T22:44:00.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2861896704 bytes 2026-02-20T22:44:00.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2884555776 bytes 2026-02-20T22:44:00.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2924715008 bytes 2026-02-20T22:44:00.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2933398528 bytes 2026-02-20T22:44:00.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2978359296 bytes 2026-02-20T22:44:01.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3026910208 bytes 2026-02-20T22:44:01.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3062927360 bytes 2026-02-20T22:44:01.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3080717312 bytes 2026-02-20T22:44:01.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3113009152 bytes 2026-02-20T22:44:01.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3121521664 bytes 2026-02-20T22:44:01.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3159165952 bytes 2026-02-20T22:44:01.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3204043776 bytes 2026-02-20T22:44:01.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3226897408 bytes 2026-02-20T22:44:01.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3264160768 bytes 2026-02-20T22:44:01.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3277978624 bytes 2026-02-20T22:44:01.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3349735424 bytes 2026-02-20T22:44:01.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3396248576 bytes 2026-02-20T22:44:01.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3427984384 bytes 2026-02-20T22:44:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3488937984 bytes 2026-02-20T22:44:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3549094912 bytes 2026-02-20T22:44:01.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3565175808 bytes 2026-02-20T22:44:01.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3601510400 bytes 2026-02-20T22:44:01.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3621366784 bytes 2026-02-20T22:44:01.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3625191424 bytes 2026-02-20T22:44:02.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3684297728 bytes 2026-02-20T22:44:02.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3731893248 bytes 2026-02-20T22:44:02.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3750523904 bytes 2026-02-20T22:44:02.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3807336448 bytes 2026-02-20T22:44:02.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3838897152 bytes 2026-02-20T22:44:02.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 3855607808 bytes 2026-02-20T22:44:02.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 3903253504 bytes 2026-02-20T22:44:02.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 3982427136 bytes 2026-02-20T22:44:02.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4045537280 bytes 2026-02-20T22:44:02.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4083380224 bytes 2026-02-20T22:44:02.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4162023424 bytes 2026-02-20T22:44:02.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4245321728 bytes 2026-02-20T22:44:02.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4250407936 bytes 2026-02-20T22:44:02.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4311436288 bytes 2026-02-20T22:44:02.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4312037376 bytes 2026-02-20T22:44:02.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4312678400 bytes 2026-02-20T22:44:02.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4320657408 bytes 2026-02-20T22:44:02.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4343106560 bytes 2026-02-20T22:44:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4359144448 bytes 2026-02-20T22:44:03.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4396084224 bytes 2026-02-20T22:44:03.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4418387968 bytes 2026-02-20T22:44:03.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4452606976 bytes 2026-02-20T22:44:03.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4496143360 bytes 2026-02-20T22:44:03.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4545754112 bytes 2026-02-20T22:44:03.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4586603520 bytes 2026-02-20T22:44:03.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4631886848 bytes 2026-02-20T22:44:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4652338176 bytes 2026-02-20T22:44:03.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4673664000 bytes 2026-02-20T22:44:03.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 4695272448 bytes 2026-02-20T22:44:03.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 4805996544 bytes 2026-02-20T22:44:03.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 4827369472 bytes 2026-02-20T22:44:03.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 4863820800 bytes 2026-02-20T22:44:03.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 4930332672 bytes 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_49 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:44:03.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_49 128M 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:03.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2026-02-20T22:44:03.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:03.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:03.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:44:03.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:44:03.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2026-02-20T22:44:04.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:44:04.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_49': 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1481a311f492 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1481a311f492 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:44:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:44:03 2026 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:44:03 2026 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:44:03 2026 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: be759c64-ef88-4e79-b56b-60211e4d3d1f 2026-02-20T22:44:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_49 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:44:04.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:44:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=13 2026-02-20T22:44:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:44:04.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:44:04.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 58436608 bytes 2026-02-20T22:44:04.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 59049984 bytes 2026-02-20T22:44:05.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 104300544 bytes 2026-02-20T22:44:05.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 169055232 bytes 2026-02-20T22:44:05.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 200187904 bytes 2026-02-20T22:44:05.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 242553856 bytes 2026-02-20T22:44:05.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 303905792 bytes 2026-02-20T22:44:05.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 362858496 bytes 2026-02-20T22:44:05.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 398215168 bytes 2026-02-20T22:44:05.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 460959744 bytes 2026-02-20T22:44:05.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 535780352 bytes 2026-02-20T22:44:05.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 544711680 bytes 2026-02-20T22:44:05.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 568926208 bytes 2026-02-20T22:44:05.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 590392320 bytes 2026-02-20T22:44:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 600366080 bytes 2026-02-20T22:44:05.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 623934464 bytes 2026-02-20T22:44:05.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 635763712 bytes 2026-02-20T22:44:05.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 691267584 bytes 2026-02-20T22:44:05.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 726552576 bytes 2026-02-20T22:44:05.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 745877504 bytes 2026-02-20T22:44:06.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 806248448 bytes 2026-02-20T22:44:06.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 806874112 bytes 2026-02-20T22:44:06.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 866110464 bytes 2026-02-20T22:44:06.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 911789056 bytes 2026-02-20T22:44:06.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 937592832 bytes 2026-02-20T22:44:06.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 972296192 bytes 2026-02-20T22:44:06.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 972918784 bytes 2026-02-20T22:44:06.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1026589696 bytes 2026-02-20T22:44:06.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1067823104 bytes 2026-02-20T22:44:06.512 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1119289344 bytes 2026-02-20T22:44:06.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1179825152 bytes 2026-02-20T22:44:06.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1228886016 bytes 2026-02-20T22:44:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1256909824 bytes 2026-02-20T22:44:06.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1271088128 bytes 2026-02-20T22:44:06.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1282379776 bytes 2026-02-20T22:44:06.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1334991872 bytes 2026-02-20T22:44:06.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1416954880 bytes 2026-02-20T22:44:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1432386560 bytes 2026-02-20T22:44:07.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1472863232 bytes 2026-02-20T22:44:07.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1479810048 bytes 2026-02-20T22:44:07.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1503390720 bytes 2026-02-20T22:44:07.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1599850496 bytes 2026-02-20T22:44:07.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1646837760 bytes 2026-02-20T22:44:07.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1681646592 bytes 2026-02-20T22:44:07.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1737310208 bytes 2026-02-20T22:44:07.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1766117376 bytes 2026-02-20T22:44:07.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1842872320 bytes 2026-02-20T22:44:07.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1896222720 bytes 2026-02-20T22:44:07.640 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1918044160 bytes 2026-02-20T22:44:07.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1989583872 bytes 2026-02-20T22:44:07.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2061788160 bytes 2026-02-20T22:44:07.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2079084544 bytes 2026-02-20T22:44:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2090081280 bytes 2026-02-20T22:44:07.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2139785216 bytes 2026-02-20T22:44:07.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2174964736 bytes 2026-02-20T22:44:08.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2200331264 bytes 2026-02-20T22:44:08.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2270337024 bytes 2026-02-20T22:44:08.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2300077056 bytes 2026-02-20T22:44:08.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2335584256 bytes 2026-02-20T22:44:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2400641024 bytes 2026-02-20T22:44:08.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2445856768 bytes 2026-02-20T22:44:08.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2509642752 bytes 2026-02-20T22:44:08.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2531305472 bytes 2026-02-20T22:44:08.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2582988800 bytes 2026-02-20T22:44:08.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2630823936 bytes 2026-02-20T22:44:08.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2678832128 bytes 2026-02-20T22:44:08.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2776987648 bytes 2026-02-20T22:44:08.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2805983232 bytes 2026-02-20T22:44:08.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2806595584 bytes 2026-02-20T22:44:08.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2857934848 bytes 2026-02-20T22:44:08.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2890092544 bytes 2026-02-20T22:44:08.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2898465792 bytes 2026-02-20T22:44:09.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2978201600 bytes 2026-02-20T22:44:09.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3053474816 bytes 2026-02-20T22:44:09.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3108552704 bytes 2026-02-20T22:44:09.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3159961600 bytes 2026-02-20T22:44:09.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3203115008 bytes 2026-02-20T22:44:09.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3213947904 bytes 2026-02-20T22:44:09.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3292982272 bytes 2026-02-20T22:44:09.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3343927296 bytes 2026-02-20T22:44:09.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3386315776 bytes 2026-02-20T22:44:09.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3436250112 bytes 2026-02-20T22:44:09.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3474360320 bytes 2026-02-20T22:44:09.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3533834240 bytes 2026-02-20T22:44:09.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3618337792 bytes 2026-02-20T22:44:09.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3663237120 bytes 2026-02-20T22:44:09.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3707781120 bytes 2026-02-20T22:44:09.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3727407104 bytes 2026-02-20T22:44:09.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3728026624 bytes 2026-02-20T22:44:10.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3773430784 bytes 2026-02-20T22:44:10.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3862635520 bytes 2026-02-20T22:44:10.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3918497792 bytes 2026-02-20T22:44:10.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3936641024 bytes 2026-02-20T22:44:10.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3948855296 bytes 2026-02-20T22:44:10.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3990212608 bytes 2026-02-20T22:44:10.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4027449344 bytes 2026-02-20T22:44:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4051572736 bytes 2026-02-20T22:44:10.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4108975104 bytes 2026-02-20T22:44:10.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4143927296 bytes 2026-02-20T22:44:10.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4167623680 bytes 2026-02-20T22:44:10.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4186173440 bytes 2026-02-20T22:44:10.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4243190784 bytes 2026-02-20T22:44:10.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4272726016 bytes 2026-02-20T22:44:10.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4311045120 bytes 2026-02-20T22:44:10.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4359127040 bytes 2026-02-20T22:44:10.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4383453184 bytes 2026-02-20T22:44:10.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4412062720 bytes 2026-02-20T22:44:10.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4439869440 bytes 2026-02-20T22:44:11.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4507186176 bytes 2026-02-20T22:44:11.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4529477632 bytes 2026-02-20T22:44:11.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4541875200 bytes 2026-02-20T22:44:11.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4592825344 bytes 2026-02-20T22:44:11.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4663440384 bytes 2026-02-20T22:44:11.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4711113728 bytes 2026-02-20T22:44:11.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4758593536 bytes 2026-02-20T22:44:11.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4803136512 bytes 2026-02-20T22:44:11.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4848505856 bytes 2026-02-20T22:44:11.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4914914304 bytes 2026-02-20T22:44:11.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5024333824 bytes 2026-02-20T22:44:11.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5032253440 bytes 2026-02-20T22:44:11.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5071833088 bytes 2026-02-20T22:44:11.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5103237120 bytes 2026-02-20T22:44:11.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5137635328 bytes 2026-02-20T22:44:11.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5177055232 bytes 2026-02-20T22:44:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5322397696 bytes 2026-02-20T22:44:12.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5347307520 bytes 2026-02-20T22:44:12.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5363164160 bytes 2026-02-20T22:44:12.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5382771712 bytes 2026-02-20T22:44:12.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5419242496 bytes 2026-02-20T22:44:12.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5443733504 bytes 2026-02-20T22:44:12.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5485550592 bytes 2026-02-20T22:44:12.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5555257344 bytes 2026-02-20T22:44:12.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5555914752 bytes 2026-02-20T22:44:12.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5595089920 bytes 2026-02-20T22:44:12.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5659706368 bytes 2026-02-20T22:44:12.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5713504256 bytes 2026-02-20T22:44:12.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5738338304 bytes 2026-02-20T22:44:12.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5756270592 bytes 2026-02-20T22:44:12.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5775055872 bytes 2026-02-20T22:44:12.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5803161600 bytes 2026-02-20T22:44:12.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5814811648 bytes 2026-02-20T22:44:12.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5815428096 bytes 2026-02-20T22:44:13.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5936459776 bytes 2026-02-20T22:44:13.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5986259968 bytes 2026-02-20T22:44:13.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5986864128 bytes 2026-02-20T22:44:13.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6011538432 bytes 2026-02-20T22:44:13.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6059601920 bytes 2026-02-20T22:44:13.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6102496256 bytes 2026-02-20T22:44:13.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6248503296 bytes 2026-02-20T22:44:13.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6268306432 bytes 2026-02-20T22:44:13.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6329304064 bytes 2026-02-20T22:44:13.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6345232384 bytes 2026-02-20T22:44:13.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6360355840 bytes 2026-02-20T22:44:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6399029248 bytes 2026-02-20T22:44:13.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6422744064 bytes 2026-02-20T22:44:13.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6493487104 bytes 2026-02-20T22:44:13.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6517604352 bytes 2026-02-20T22:44:13.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6556356608 bytes 2026-02-20T22:44:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6604542976 bytes 2026-02-20T22:44:14.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6728382464 bytes 2026-02-20T22:44:14.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6833651712 bytes 2026-02-20T22:44:14.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6855645184 bytes 2026-02-20T22:44:14.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6931205120 bytes 2026-02-20T22:44:14.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6964480000 bytes 2026-02-20T22:44:14.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6985877504 bytes 2026-02-20T22:44:14.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7034996736 bytes 2026-02-20T22:44:14.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7085983744 bytes 2026-02-20T22:44:14.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7120202752 bytes 2026-02-20T22:44:14.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7158619136 bytes 2026-02-20T22:44:14.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7242447872 bytes 2026-02-20T22:44:14.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7262673920 bytes 2026-02-20T22:44:14.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7334055936 bytes 2026-02-20T22:44:14.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7367157760 bytes 2026-02-20T22:44:14.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7394436096 bytes 2026-02-20T22:44:15.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7463671808 bytes 2026-02-20T22:44:15.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7499363328 bytes 2026-02-20T22:44:15.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7514482688 bytes 2026-02-20T22:44:15.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7553449984 bytes 2026-02-20T22:44:15.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7597272064 bytes 2026-02-20T22:44:15.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7658555392 bytes 2026-02-20T22:44:15.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7674487808 bytes 2026-02-20T22:44:15.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7767928832 bytes 2026-02-20T22:44:15.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7830653952 bytes 2026-02-20T22:44:15.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7865802752 bytes 2026-02-20T22:44:15.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7866408960 bytes 2026-02-20T22:44:15.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7939288064 bytes 2026-02-20T22:44:15.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7967292416 bytes 2026-02-20T22:44:15.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7972231168 bytes 2026-02-20T22:44:15.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8011556864 bytes 2026-02-20T22:44:15.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8039984128 bytes 2026-02-20T22:44:15.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8099284992 bytes 2026-02-20T22:44:16.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8144958464 bytes 2026-02-20T22:44:16.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8196783104 bytes 2026-02-20T22:44:16.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8291550208 bytes 2026-02-20T22:44:16.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8325405696 bytes 2026-02-20T22:44:16.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8417581056 bytes 2026-02-20T22:44:16.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8442226688 bytes 2026-02-20T22:44:16.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8479973376 bytes 2026-02-20T22:44:16.503 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8503341056 bytes 2026-02-20T22:44:16.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8516572160 bytes 2026-02-20T22:44:16.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8614418432 bytes 2026-02-20T22:44:16.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8656308224 bytes 2026-02-20T22:44:16.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8704881664 bytes 2026-02-20T22:44:16.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8731948032 bytes 2026-02-20T22:44:16.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8812463104 bytes 2026-02-20T22:44:16.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8818049024 bytes 2026-02-20T22:44:17.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8839967744 bytes 2026-02-20T22:44:17.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8858429440 bytes 2026-02-20T22:44:17.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8919930880 bytes 2026-02-20T22:44:17.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8939157504 bytes 2026-02-20T22:44:17.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8985241600 bytes 2026-02-20T22:44:17.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9012125696 bytes 2026-02-20T22:44:17.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9058302976 bytes 2026-02-20T22:44:17.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9085402112 bytes 2026-02-20T22:44:17.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9089453056 bytes 2026-02-20T22:44:17.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9121445888 bytes 2026-02-20T22:44:17.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9172842496 bytes 2026-02-20T22:44:17.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9204146176 bytes 2026-02-20T22:44:17.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9256561664 bytes 2026-02-20T22:44:17.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9313228800 bytes 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_50 2026-02-20T22:44:17.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n snapshot ']' 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_50 128M 2026-02-20T22:44:17.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 128M ']' 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=128M 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:44:17.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2026-02-20T22:44:17.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ is_pool_mirror_mode_image mirror 2026-02-20T22:44:17.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:17.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' image = image ']' 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:44:17.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2026-02-20T22:44:18.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:44:18.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_50': 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 148af81a17d8 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.148af81a17d8 2026-02-20T22:44:18.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:44:17 2026 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:44:17 2026 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:44:17 2026 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7389bdfb-9491-4580-9c72-b6dfcd587c3b 2026-02-20T22:44:18.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n '' ']' 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_50 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:44:18.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:44:18.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T22:44:18.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:44:18.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:44:18.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 48954368 bytes 2026-02-20T22:44:19.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 102267904 bytes 2026-02-20T22:44:19.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 116170752 bytes 2026-02-20T22:44:19.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 159558656 bytes 2026-02-20T22:44:19.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 206121984 bytes 2026-02-20T22:44:19.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 231419904 bytes 2026-02-20T22:44:19.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 282147840 bytes 2026-02-20T22:44:19.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 371108864 bytes 2026-02-20T22:44:19.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 417074176 bytes 2026-02-20T22:44:19.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 449632256 bytes 2026-02-20T22:44:19.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 461804544 bytes 2026-02-20T22:44:19.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 565341184 bytes 2026-02-20T22:44:19.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 592523264 bytes 2026-02-20T22:44:19.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 626359296 bytes 2026-02-20T22:44:19.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 694012928 bytes 2026-02-20T22:44:19.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 749788160 bytes 2026-02-20T22:44:19.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 812716032 bytes 2026-02-20T22:44:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 883554304 bytes 2026-02-20T22:44:20.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 965761024 bytes 2026-02-20T22:44:20.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1010772992 bytes 2026-02-20T22:44:20.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1088099328 bytes 2026-02-20T22:44:20.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1088716800 bytes 2026-02-20T22:44:20.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1132359680 bytes 2026-02-20T22:44:20.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1187377152 bytes 2026-02-20T22:44:20.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1286294528 bytes 2026-02-20T22:44:20.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1322262528 bytes 2026-02-20T22:44:20.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1385450496 bytes 2026-02-20T22:44:20.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1435824128 bytes 2026-02-20T22:44:20.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1517443072 bytes 2026-02-20T22:44:20.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1623106560 bytes 2026-02-20T22:44:20.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1665280000 bytes 2026-02-20T22:44:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1712305152 bytes 2026-02-20T22:44:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1761555456 bytes 2026-02-20T22:44:21.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1816818688 bytes 2026-02-20T22:44:21.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1838599168 bytes 2026-02-20T22:44:21.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1860781056 bytes 2026-02-20T22:44:21.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1870481408 bytes 2026-02-20T22:44:21.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1926772736 bytes 2026-02-20T22:44:21.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1944805376 bytes 2026-02-20T22:44:21.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 2002300928 bytes 2026-02-20T22:44:21.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 2037310464 bytes 2026-02-20T22:44:21.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2111877120 bytes 2026-02-20T22:44:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2187712512 bytes 2026-02-20T22:44:21.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2221252608 bytes 2026-02-20T22:44:21.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2282077184 bytes 2026-02-20T22:44:21.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2323612672 bytes 2026-02-20T22:44:21.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2395334656 bytes 2026-02-20T22:44:21.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2508208128 bytes 2026-02-20T22:44:21.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2578039808 bytes 2026-02-20T22:44:22.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2685855744 bytes 2026-02-20T22:44:22.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2698980352 bytes 2026-02-20T22:44:22.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2761333760 bytes 2026-02-20T22:44:22.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2761956352 bytes 2026-02-20T22:44:22.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2785482752 bytes 2026-02-20T22:44:22.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2833941504 bytes 2026-02-20T22:44:22.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2866335744 bytes 2026-02-20T22:44:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2935486464 bytes 2026-02-20T22:44:22.504 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2986472448 bytes 2026-02-20T22:44:22.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 3000600576 bytes 2026-02-20T22:44:22.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 3077246976 bytes 2026-02-20T22:44:22.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 3126682624 bytes 2026-02-20T22:44:22.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 3133595648 bytes 2026-02-20T22:44:22.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 3189545984 bytes 2026-02-20T22:44:22.825 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3233713152 bytes 2026-02-20T22:44:22.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3317341184 bytes 2026-02-20T22:44:22.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3333958656 bytes 2026-02-20T22:44:22.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3353479168 bytes 2026-02-20T22:44:23.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3364505600 bytes 2026-02-20T22:44:23.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3405165568 bytes 2026-02-20T22:44:23.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3468272640 bytes 2026-02-20T22:44:23.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3480368128 bytes 2026-02-20T22:44:23.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3497897984 bytes 2026-02-20T22:44:23.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3552707584 bytes 2026-02-20T22:44:23.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3564850176 bytes 2026-02-20T22:44:23.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3597719552 bytes 2026-02-20T22:44:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3630195712 bytes 2026-02-20T22:44:23.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3681787904 bytes 2026-02-20T22:44:23.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3724036096 bytes 2026-02-20T22:44:23.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3759596544 bytes 2026-02-20T22:44:23.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3819022336 bytes 2026-02-20T22:44:23.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3819619328 bytes 2026-02-20T22:44:23.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3822267392 bytes 2026-02-20T22:44:23.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3872491520 bytes 2026-02-20T22:44:23.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3902520320 bytes 2026-02-20T22:44:23.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3927159808 bytes 2026-02-20T22:44:23.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3945702400 bytes 2026-02-20T22:44:23.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3980315648 bytes 2026-02-20T22:44:24.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4017372160 bytes 2026-02-20T22:44:24.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4127438848 bytes 2026-02-20T22:44:24.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4154591232 bytes 2026-02-20T22:44:24.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4198118400 bytes 2026-02-20T22:44:24.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4277783552 bytes 2026-02-20T22:44:24.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4278405120 bytes 2026-02-20T22:44:24.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4318616576 bytes 2026-02-20T22:44:24.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4348147712 bytes 2026-02-20T22:44:24.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4396855296 bytes 2026-02-20T22:44:24.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4397470720 bytes 2026-02-20T22:44:24.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4440501248 bytes 2026-02-20T22:44:24.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4472423424 bytes 2026-02-20T22:44:24.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4533494784 bytes 2026-02-20T22:44:24.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4579624960 bytes 2026-02-20T22:44:24.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4625441792 bytes 2026-02-20T22:44:24.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:44:24.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:44:24.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:44:24.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_images cluster2 mirror 50 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_count=50 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local count 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_count=0 2026-02-20T22:44:24.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:44:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:44:24.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:44:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2026-02-20T22:44:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'images: ' 2026-02-20T22:44:24.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 2 2026-02-20T22:44:24.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:44:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ count=50 2026-02-20T22:44:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 50 = 50 2026-02-20T22:44:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:24.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_healthy cluster2 mirror 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:44:24.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:24.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:44:24.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2026-02-20T22:44:24.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'image health:' 2026-02-20T22:44:24.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 3 2026-02-20T22:44:24.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:44:24.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=OK 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = ERROR 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = OK 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_images cluster1 mirror 50 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_count=50 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local count 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_count=0 2026-02-20T22:44:24.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:44:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:44:24.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:44:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:44:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'images: ' 2026-02-20T22:44:24.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 2 2026-02-20T22:44:24.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ count=50 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 50 = 50 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_healthy cluster1 mirror 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:44:24.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:44:24.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:44:24.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state 2026-02-20T22:44:24.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:44:24.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:24.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:44:24.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:44:24.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'image health:' 2026-02-20T22:44:24.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 3 2026-02-20T22:44:24.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:44:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=WARNING 2026-02-20T22:44:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test WARNING = ERROR 2026-02-20T22:44:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test WARNING = OK 2026-02-20T22:44:24.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 2 -ne 1 2026-02-20T22:44:24.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 30 2026-02-20T22:44:37.701 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.41055556134312277) > weight (0.25) 2026-02-20T22:44:37.701 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7965059779380131) > weight (0.25) 2026-02-20T22:44:37.702 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-20T22:44:37.702 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-20T22:44:37.702 INFO:tasks.rbd_mirror_thrash:waiting for 6.110594002225891 secs before reviving daemons 2026-02-20T22:44:37.703 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:44:37.692+0000 7f99236bd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 19654) UID: 1000 2026-02-20T22:44:43.813 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-20T22:44:43.813 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:44:43.814 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-20T22:44:43.814 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-20T22:44:43.814 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-20T22:44:43.815 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-20T22:44:43.815 INFO:tasks.rbd_mirror_thrash:waiting for 102.03561501806408 secs before thrashing 2026-02-20T22:44:54.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:44:54.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'image health:' 2026-02-20T22:44:54.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 3 2026-02-20T22:44:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:44:55.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=WARNING 2026-02-20T22:44:55.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test WARNING = ERROR 2026-02-20T22:44:55.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test WARNING = OK 2026-02-20T22:44:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:44:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 3 -ne 1 2026-02-20T22:44:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 30 2026-02-20T22:45:25.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:45:25.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'image health:' 2026-02-20T22:45:25.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 3 2026-02-20T22:45:25.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=OK 2026-02-20T22:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = ERROR 2026-02-20T22:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = OK 2026-02-20T22:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:25.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: compare many images' 2026-02-20T22:45:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:45:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:45:25.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:45:25 TEST: compare many images 2026-02-20T22:45:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:45:25 TEST: compare many images 2026-02-20T22:45:25.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 50 2026-02-20T22:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_1 2026-02-20T22:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_1 snap 2026-02-20T22:45:25.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:26.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:26.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2026-02-20T22:45:27.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:45:27.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:45:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:45:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:27.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:27.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:27.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:27.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:27.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2026-02-20T22:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 1 closest matches: 2026-02-20T22:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_19 2026-02-20T22:45:28.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_1 2026-02-20T22:45:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:45:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:28.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:45:28.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:28.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:45:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:45:28.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:45:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_1 2026-02-20T22:45:28.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 3 closest matches: 2026-02-20T22:45:28.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_11 2026-02-20T22:45:28.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_12 2026-02-20T22:45:28.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_18 2026-02-20T22:45:28.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_1 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:45:28.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:28.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:45:28.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:45:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_1 2026-02-20T22:45:28.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 4 closest matches: 2026-02-20T22:45:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_13 2026-02-20T22:45:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_14 2026-02-20T22:45:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_15 2026-02-20T22:45:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_16 2026-02-20T22:45:28.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_1 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:45:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:28.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:45:28.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:45:28.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:28.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:28.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:28.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_1 2026-02-20T22:45:28.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:45:28.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:45:28.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_1 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:28.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:28.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2026-02-20T22:45:28.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 86 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_1 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:28.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_1.status 2026-02-20T22:45:28.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_1.status 2026-02-20T22:45:28.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:28.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.status 2026-02-20T22:45:28.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_1.status 2026-02-20T22:45:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_1.status 2026-02-20T22:45:28.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.status 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.status 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2026-02-20T22:45:28.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:28.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:45:28.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=86 2026-02-20T22:45:28.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:28.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:28.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:29.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2026-02-20T22:45:29.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:29.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=35 2026-02-20T22:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 86 = 35 2026-02-20T22:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:29.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:45:29.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2026-02-20T22:45:29.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:29.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:29.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=35 2026-02-20T22:45:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 86 = 35 2026-02-20T22:45:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:29.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:45:30.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2026-02-20T22:45:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:30.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:30.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:30.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:45:30.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:45:30.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:45:30.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:45:30.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 86 = '' 2026-02-20T22:45:30.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:30.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:45:32.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:32.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:32.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:32.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:32.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.status 2026-02-20T22:45:32.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2026-02-20T22:45:32.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:32.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=86 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 86 = 86 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:32.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:45:32.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:33.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2026-02-20T22:45:33.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_1': 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10a7299ffa87 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10a7299ffa87 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:45:33.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:35:52 2026 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:35:52 2026 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:35:52 2026 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3ec1d56b-1fff-4c3b-a39e-7c1c401a9ca7 2026-02-20T22:45:33.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:45:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:33.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_1 snap 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.export 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.export 2026-02-20T22:45:33.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.export 2026-02-20T22:45:33.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.export 2026-02-20T22:45:33.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:33.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.export 2026-02-20T22:45:33.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:33.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.export 2026-02-20T22:45:33.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_1.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_1.export 2026-02-20T22:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_2 2026-02-20T22:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_2 snap 2026-02-20T22:45:33.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:33.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:33.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:33.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:33.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:45:34.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:34.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:45:35.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:35.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:45:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:35.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:35.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:35.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:35.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:35.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:35.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:35.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:35.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2026-02-20T22:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:45:36.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:45:36.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:45:36.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:36.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2026-02-20T22:45:36.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 88 2026-02-20T22:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_2 2026-02-20T22:45:36.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_2.status 2026-02-20T22:45:36.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_2.status 2026-02-20T22:45:36.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:36.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.status 2026-02-20T22:45:36.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_2.status 2026-02-20T22:45:36.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_2.status 2026-02-20T22:45:36.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:36.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.status 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.status 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2026-02-20T22:45:36.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:45:36.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=88 2026-02-20T22:45:36.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:36.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:36.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:45:37.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:37.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:37.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:37.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:37.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:37.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2026-02-20T22:45:37.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:37.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=36 2026-02-20T22:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 88 = 36 2026-02-20T22:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:37.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:45:37.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2026-02-20T22:45:37.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:37.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:37.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=36 2026-02-20T22:45:37.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 88 = 36 2026-02-20T22:45:37.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:37.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:45:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:38.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2026-02-20T22:45:38.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:38.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:38.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:45:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:45:38.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:45:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:45:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 88 = '' 2026-02-20T22:45:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:38.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:45:39.974 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-20T22:45:39.974 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-20T22:45:39.974 INFO:tasks.rbd_mirror_thrash:waiting for 5.274012906244945 secs before reviving daemons 2026-02-20T22:45:39.975 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:45:39.967+0000 7f09f273b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 29287) UID: 1000 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.status 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2026-02-20T22:45:40.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:40.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=88 2026-02-20T22:45:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 88 = 88 2026-02-20T22:45:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2026-02-20T22:45:40.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:45:40.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:41.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_2': 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10af1b717e00 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10af1b717e00 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:45:41.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:07 2026 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:07 2026 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:07 2026 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 241a16d7-c45c-4a2f-9678-2850dc15e21b 2026-02-20T22:45:41.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:45:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:41.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_2 snap 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.export 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.export 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.export 2026-02-20T22:45:41.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.export 2026-02-20T22:45:41.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:41.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.export 2026-02-20T22:45:41.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:41.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.export 2026-02-20T22:45:41.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_2.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_2.export 2026-02-20T22:45:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_3 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_3 snap 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:41.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:45:42.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2026-02-20T22:45:42.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:42.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2026-02-20T22:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:43.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:45:43.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:43.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:43.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:43.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:43.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:43.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:43.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2026-02-20T22:45:44.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:45:44.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:45:44.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:45:44.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:45:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:44.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2026-02-20T22:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 2 closest matches: 2026-02-20T22:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_33 2026-02-20T22:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_37 2026-02-20T22:45:44.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:44.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:44.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:45:44.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:45:44.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:44.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:44.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:44.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2026-02-20T22:45:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 3 closest matches: 2026-02-20T22:45:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_31 2026-02-20T22:45:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_35 2026-02-20T22:45:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_39 2026-02-20T22:45:44.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2026-02-20T22:45:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:45:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:44.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:44.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:44.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:45:44.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:45:44.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:45:44.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:44.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2026-02-20T22:45:44.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:45:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:45:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:45:44.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:45:44.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:45:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:45:44.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_3 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:45:44.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2026-02-20T22:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:44.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2026-02-20T22:45:44.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 90 2026-02-20T22:45:44.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_3 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:44.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_3.status 2026-02-20T22:45:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_3.status 2026-02-20T22:45:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:44.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.status 2026-02-20T22:45:44.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_3.status 2026-02-20T22:45:44.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_3.status 2026-02-20T22:45:44.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.status 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.status 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2026-02-20T22:45:44.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:44.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:45:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=90 2026-02-20T22:45:44.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:45:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:45.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2026-02-20T22:45:45.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:45.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=37 2026-02-20T22:45:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 90 = 37 2026-02-20T22:45:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:45.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:45:45.249 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-20T22:45:45.249 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:45:45.249 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-20T22:45:45.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-20T22:45:45.249 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-20T22:45:45.251 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-20T22:45:45.251 INFO:tasks.rbd_mirror_thrash:waiting for 88.88045478508342 secs before thrashing 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2026-02-20T22:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:45.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=37 2026-02-20T22:45:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 90 = 37 2026-02-20T22:45:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:45.671 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:46.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2026-02-20T22:45:46.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:46.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:46.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:45:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:45:46.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 90 = '' 2026-02-20T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:46.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:45:46.728 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:45:46.723+0000 7f72ea4a6640 -1 reset not still connected to 0x55c81443a0d0 2026-02-20T22:45:46.730 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:45:46.723+0000 7f72ea4a6640 -1 reset not still connected to 0x55c85545e9c0 2026-02-20T22:45:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:48.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.status 2026-02-20T22:45:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2026-02-20T22:45:48.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:48.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=90 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 90 = 90 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:48.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:45:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:49.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_3': 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10da35995fee 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10da35995fee 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:22 2026 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:22 2026 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:22 2026 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2e6164f9-d122-4af4-b228-4bf39a22c6fb 2026-02-20T22:45:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:45:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_3 snap 2026-02-20T22:45:49.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.export 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.export 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.export 2026-02-20T22:45:49.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.export 2026-02-20T22:45:49.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:49.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.export 2026-02-20T22:45:49.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:49.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.export 2026-02-20T22:45:49.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_3.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_3.export 2026-02-20T22:45:49.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:49.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:45:49.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_4 2026-02-20T22:45:49.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_4 snap 2026-02-20T22:45:49.620 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:49.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:49.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:49.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:49.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:45:50.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:45:50.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:50.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:50.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:51.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:45:51.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:51.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:45:51.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:51.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:51.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:51.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:51.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:51.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:51.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:51.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:51.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2026-02-20T22:45:51.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_4'] 2026-02-20T22:45:51.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:45:51.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:51.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:51.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:51.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:45:51.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:45:51.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:51.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2026-02-20T22:45:51.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:45:51.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:45:51.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:45:51.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:45:51.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_4 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:51.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:51.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2026-02-20T22:45:52.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 92 2026-02-20T22:45:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_4 2026-02-20T22:45:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:45:52.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:45:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:45:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_4.status 2026-02-20T22:45:52.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_4.status 2026-02-20T22:45:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:52.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.status 2026-02-20T22:45:52.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_4.status 2026-02-20T22:45:52.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_4.status 2026-02-20T22:45:52.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:45:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.status 2026-02-20T22:45:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:52.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.status 2026-02-20T22:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2026-02-20T22:45:52.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:52.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:45:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=92 2026-02-20T22:45:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:45:52.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:52.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2026-02-20T22:45:52.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:52.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:52.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:52.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:45:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:45:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:45:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:45:52.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 92 = '' 2026-02-20T22:45:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:52.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:45:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:53.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:53.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2026-02-20T22:45:53.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:53.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:45:53.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:45:53.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:45:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:45:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 92 = '' 2026-02-20T22:45:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:45:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:45:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:54.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:54.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.status 2026-02-20T22:45:54.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2026-02-20T22:45:54.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:45:54.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:45:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=92 2026-02-20T22:45:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 92 = 92 2026-02-20T22:45:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:45:54.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:55.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_4': 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 121392667ca5 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.121392667ca5 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:45:55.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:34 2026 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:34 2026 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:34 2026 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ddee4f8b-4aff-4c02-80d5-06e133a89824 2026-02-20T22:45:55.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:45:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_4 snap 2026-02-20T22:45:55.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.export 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.export 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.export 2026-02-20T22:45:55.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.export 2026-02-20T22:45:55.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:55.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.export 2026-02-20T22:45:55.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:45:55.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.export 2026-02-20T22:45:55.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_4.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_4.export 2026-02-20T22:45:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:45:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:45:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_5 2026-02-20T22:45:55.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_5 snap 2026-02-20T22:45:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:45:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:45:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:45:55.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:57.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:45:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:45:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:45:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:57.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:45:58.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:45:58.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:45:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:45:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:58.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:58.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:58.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:58.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:58.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:58.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:58.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:58.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:58.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-20T22:45:58.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:45:58.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:45:58.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:58.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:58.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:45:58.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:45:58.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:58.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:58.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:45:58.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-20T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:45:58.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:58.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:45:58.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:58.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:58.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:45:58.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:45:58.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:45:58.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:45:58.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2026-02-20T22:45:58.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:45:58.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:45:58.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:58.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:58.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2026-02-20T22:45:58.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:58.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:58.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:45:58.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:45:58.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:58.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:58.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:45:58.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2026-02-20T22:45:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:45:58.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:45:58.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:45:58.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:45:58.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 1 2026-02-20T22:45:59.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:45:59.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:45:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:45:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-20T22:45:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:45:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:45:59.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:59.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:45:59.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:45:59.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:45:59.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:45:59.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:45:59.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:45:59.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:59.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:45:59.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-20T22:46:00.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:00.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:00.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:00.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:00.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:00.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:00.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-20T22:46:00.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2026-02-20T22:46:00.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:00.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:00.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:00.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:00.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:00.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:00.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:00.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:00.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:00.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2026-02-20T22:46:00.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:00.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:00.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:46:00.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:00.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:00.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:00.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:00.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:46:00.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:46:00.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:46:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:00.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:00.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:00.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2026-02-20T22:46:00.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:00.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:00.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:00.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:00.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 2 2026-02-20T22:46:02.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:02.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:02.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:02.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:02.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:02.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:02.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:02.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:02.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-20T22:46:02.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:02.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:02.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:02.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:02.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-20T22:46:02.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:02.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:02.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-20T22:46:02.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:02.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:02.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:02.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:02.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:02.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:02.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:02.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:02.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:02.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2026-02-20T22:46:02.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:02.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:02.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:46:02.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:46:02.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:46:02.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:46:02.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:46:02.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:02.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:02.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:02.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2026-02-20T22:46:02.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:02.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:02.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:02.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:02.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 4 2026-02-20T22:46:06.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:06.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-20T22:46:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:06.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:06.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:06.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:06.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:06.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:06.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-20T22:46:06.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:06.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:06.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-20T22:46:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:06.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:06.636 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:06.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:06.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:06.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:06.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-20T22:46:06.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:06.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:06.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:06.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:06.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:06.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:06.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:06.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2026-02-20T22:46:06.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:06.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:06.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:06.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:06.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:06.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:46:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:46:06.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:46:06.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:06.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2026-02-20T22:46:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:06.850 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:06.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 8 2026-02-20T22:46:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:14.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:14.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:14.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:14.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:14.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:14.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:14.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:14.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2026-02-20T22:46:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:no valid command found; 1 closest matches: 2026-02-20T22:46:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status mirror/image_50 2026-02-20T22:46:14.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:14.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:14.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:14.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:14.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:14.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:14.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:14.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2026-02-20T22:46:15.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2026-02-20T22:46:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:15.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:15.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:15.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:15.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:15.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:15.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2026-02-20T22:46:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2026-02-20T22:46:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:15.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:15.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:15.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:46:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:46:15.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:46:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:46:15.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2026-02-20T22:46:15.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:15.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:15.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_5 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:15.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2026-02-20T22:46:15.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 94 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_5 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:15.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_5.status 2026-02-20T22:46:15.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_5.status 2026-02-20T22:46:15.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:15.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.status 2026-02-20T22:46:15.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_5.status 2026-02-20T22:46:15.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_5.status 2026-02-20T22:46:15.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.status 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.status 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2026-02-20T22:46:15.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:15.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=94 2026-02-20T22:46:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:15.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:15.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:15.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2026-02-20T22:46:15.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:15.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:15.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:15.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:15.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:15.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:15.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:15.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 94 = '' 2026-02-20T22:46:15.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:15.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:16.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2026-02-20T22:46:16.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:16.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:16.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:16.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:16.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:16.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:16.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:16.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 94 = '' 2026-02-20T22:46:16.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:16.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:17.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.status 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2026-02-20T22:46:17.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:17.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=94 2026-02-20T22:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 94 = 94 2026-02-20T22:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2026-02-20T22:46:17.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:17.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:18.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_5': 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10da921e5dd1 2026-02-20T22:46:18.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10da921e5dd1 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:42 2026 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:42 2026 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:42 2026 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 6be28159-a310-434e-99b8-33aa6cef5760 2026-02-20T22:46:18.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_5 snap 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.export 2026-02-20T22:46:18.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.export 2026-02-20T22:46:18.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.export 2026-02-20T22:46:18.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.export 2026-02-20T22:46:18.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:18.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.export 2026-02-20T22:46:18.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:18.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.export 2026-02-20T22:46:18.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_5.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_5.export 2026-02-20T22:46:18.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_6 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_6 snap 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:18.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2026-02-20T22:46:20.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:20.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:21.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:21.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:21.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:21.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:21.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:21.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:21.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2026-02-20T22:46:21.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:21.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:21.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:21.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:21.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_6 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:21.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:21.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2026-02-20T22:46:22.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 96 2026-02-20T22:46:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_6 2026-02-20T22:46:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:22.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_6.status 2026-02-20T22:46:22.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_6.status 2026-02-20T22:46:22.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:22.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.status 2026-02-20T22:46:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_6.status 2026-02-20T22:46:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_6.status 2026-02-20T22:46:22.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.status 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.status 2026-02-20T22:46:22.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2026-02-20T22:46:22.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=96 2026-02-20T22:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:22.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:22.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2026-02-20T22:46:22.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:22.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:22.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:22.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:22.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 96 = '' 2026-02-20T22:46:22.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:22.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:23.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:23.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2026-02-20T22:46:23.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:23.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:23.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:23.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:23.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 96 = '' 2026-02-20T22:46:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:23.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:24.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:24.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.status 2026-02-20T22:46:24.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2026-02-20T22:46:24.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:24.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:24.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=96 2026-02-20T22:46:24.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 96 = 96 2026-02-20T22:46:24.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:24.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2026-02-20T22:46:24.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:24.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:25.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_6': 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249b48eb61d 2026-02-20T22:46:25.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249b48eb61d 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:48 2026 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:48 2026 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:48 2026 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:25.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1b45a9ef-3d84-4767-b2e5-fe95a04ac4ac 2026-02-20T22:46:25.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_6 snap 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.export 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.export 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.export 2026-02-20T22:46:25.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.export 2026-02-20T22:46:25.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:25.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.export 2026-02-20T22:46:25.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:25.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.export 2026-02-20T22:46:25.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_6.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_6.export 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_7 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_7 snap 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:25.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:25.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.999471336631569) > weight (0.25) 2026-02-20T22:46:25.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.41629588923606464) > weight (0.25) 2026-02-20T22:46:25.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.3797862140849332) > weight (0.25) 2026-02-20T22:46:25.852 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.37240759975256243) > weight (0.25) 2026-02-20T22:46:25.852 INFO:tasks.rbd_mirror_thrash:waiting for 85.5622179721997 secs before thrashing 2026-02-20T22:46:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2026-02-20T22:46:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:27.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:27.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:28.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:28.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:28.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:28.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:28.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:28.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:28.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2026-02-20T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_7'] 2026-02-20T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:28.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:28.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_7 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:28.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:28.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:28.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:28.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:28.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:28.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_7 2026-02-20T22:46:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:28.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:28.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:28.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_7 2026-02-20T22:46:28.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:28.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2026-02-20T22:46:29.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 98 2026-02-20T22:46:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_7 2026-02-20T22:46:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:29.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_7.status 2026-02-20T22:46:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_7.status 2026-02-20T22:46:29.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.status 2026-02-20T22:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_7.status 2026-02-20T22:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_7.status 2026-02-20T22:46:29.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.status 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.status 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2026-02-20T22:46:29.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:29.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=98 2026-02-20T22:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:29.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:29.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2026-02-20T22:46:29.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:30.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:30.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:30.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:30.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:30.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:30.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:30.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 98 = '' 2026-02-20T22:46:30.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:30.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2026-02-20T22:46:30.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:30.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:30.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:30.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:30.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 98 = '' 2026-02-20T22:46:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:30.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:31.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.status 2026-02-20T22:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2026-02-20T22:46:31.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:31.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=98 2026-02-20T22:46:31.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 98 = 98 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:31.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:32.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2026-02-20T22:46:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_7': 2026-02-20T22:46:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:32.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12132a85a529 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12132a85a529 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:36:59 2026 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:36:59 2026 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:36:59 2026 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9795c60e-64e0-4de7-a6fb-f8429e83e276 2026-02-20T22:46:32.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_7 snap 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.export 2026-02-20T22:46:32.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.export 2026-02-20T22:46:32.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.export 2026-02-20T22:46:32.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.export 2026-02-20T22:46:32.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:32.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.export 2026-02-20T22:46:32.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:32.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.export 2026-02-20T22:46:32.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_7.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_7.export 2026-02-20T22:46:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_8 2026-02-20T22:46:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_8 snap 2026-02-20T22:46:32.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:32.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:32.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:32.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:32.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:34.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:34.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:35.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2026-02-20T22:46:35.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:35.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:35.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:35.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:35.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:35.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:35.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:35.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:35.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:35.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2026-02-20T22:46:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-20T22:46:35.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:35.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:35.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_8 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:35.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:35.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:35.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:35.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:35.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_8 2026-02-20T22:46:35.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_8'] 2026-02-20T22:46:35.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:35.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_8 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:35.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:35.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:35.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:35.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:35.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:35.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:35.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_8 2026-02-20T22:46:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:35.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:35.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:35.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:35.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:35.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:35.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:35.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_8 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:35.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:35.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:35.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2026-02-20T22:46:36.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 100 2026-02-20T22:46:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_8 2026-02-20T22:46:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:36.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_8.status 2026-02-20T22:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_8.status 2026-02-20T22:46:36.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:36.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.status 2026-02-20T22:46:36.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_8.status 2026-02-20T22:46:36.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_8.status 2026-02-20T22:46:36.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.status 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.status 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2026-02-20T22:46:36.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:36.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=100 2026-02-20T22:46:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:36.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:36.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2026-02-20T22:46:36.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:37.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=42 2026-02-20T22:46:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 100 = 42 2026-02-20T22:46:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:37.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:37.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2026-02-20T22:46:37.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:37.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:37.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=42 2026-02-20T22:46:37.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 100 = 42 2026-02-20T22:46:37.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:37.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:38.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:38.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:38.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2026-02-20T22:46:38.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:38.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:38.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:38.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:38.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:38.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 100 = '' 2026-02-20T22:46:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:38.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.status 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2026-02-20T22:46:39.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:39.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=100 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 100 = 100 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:39.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:40.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2026-02-20T22:46:40.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_8': 2026-02-20T22:46:40.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249806f97ff 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249806f97ff 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:06 2026 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:06 2026 2026-02-20T22:46:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:06 2026 2026-02-20T22:46:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 25965088-a176-4775-96aa-3e7b420cb588 2026-02-20T22:46:40.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_8 snap 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:40.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.export 2026-02-20T22:46:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.export 2026-02-20T22:46:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.export 2026-02-20T22:46:40.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.export 2026-02-20T22:46:41.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:41.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.export 2026-02-20T22:46:41.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:41.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.export 2026-02-20T22:46:41.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_8.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_8.export 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_9 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_9 snap 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:41.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:42.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2026-02-20T22:46:43.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:43.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:43.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:43.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:43.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:43.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:43.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:43.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:43.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:43.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:43.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:43.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:43.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2026-02-20T22:46:43.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-20T22:46:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:43.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:43.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:43.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:43.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:43.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:43.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2026-02-20T22:46:43.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2026-02-20T22:46:43.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:43.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:43.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_9 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:43.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:46:43.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:46:43.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:46:43.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:43.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:43.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:46:43.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_9 2026-02-20T22:46:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_9 2026-02-20T22:46:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2026-02-20T22:46:44.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 102 2026-02-20T22:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_9 2026-02-20T22:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:44.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_9.status 2026-02-20T22:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_9.status 2026-02-20T22:46:44.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.status 2026-02-20T22:46:44.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_9.status 2026-02-20T22:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_9.status 2026-02-20T22:46:44.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.status 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.status 2026-02-20T22:46:44.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2026-02-20T22:46:44.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:44.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=102 2026-02-20T22:46:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:44.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:44.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:44.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:44.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2026-02-20T22:46:44.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:45.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=43 2026-02-20T22:46:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 102 = 43 2026-02-20T22:46:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:45.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:45.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2026-02-20T22:46:45.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:45.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:45.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=43 2026-02-20T22:46:45.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 102 = 43 2026-02-20T22:46:45.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:45.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:46.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2026-02-20T22:46:46.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:46.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:46.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:46.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:46.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 102 = '' 2026-02-20T22:46:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:46.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.status 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2026-02-20T22:46:47.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:47.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=102 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 102 = 102 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:47.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:47.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:48.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_9': 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10afdc48e07a 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10afdc48e07a 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:18 2026 2026-02-20T22:46:48.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:18 2026 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:18 2026 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: eae8172c-87a1-437f-8f5e-44b17669fdc4 2026-02-20T22:46:48.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_9 snap 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.export 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.export 2026-02-20T22:46:48.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.export 2026-02-20T22:46:48.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.export 2026-02-20T22:46:49.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:49.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.export 2026-02-20T22:46:49.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:49.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.export 2026-02-20T22:46:49.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_9.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_9.export 2026-02-20T22:46:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:49.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_10 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_10 snap 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:50.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2026-02-20T22:46:50.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:50.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:50.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:50.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:51.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:51.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:51.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2026-02-20T22:46:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:51.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:51.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:51.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:51.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:51.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2026-02-20T22:46:51.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:51.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:51.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:51.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_10 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:51.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:51.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2026-02-20T22:46:52.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 104 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_10 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:52.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_10.status 2026-02-20T22:46:52.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_10.status 2026-02-20T22:46:52.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:52.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.status 2026-02-20T22:46:52.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_10.status 2026-02-20T22:46:52.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_10.status 2026-02-20T22:46:52.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.status 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.status 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2026-02-20T22:46:52.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:52.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:46:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=104 2026-02-20T22:46:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:52.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2026-02-20T22:46:53.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:53.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:53.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=44 2026-02-20T22:46:53.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 104 = 44 2026-02-20T22:46:53.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:53.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2026-02-20T22:46:53.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:53.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:53.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=44 2026-02-20T22:46:53.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 104 = 44 2026-02-20T22:46:53.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:53.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:46:54.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2026-02-20T22:46:54.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:54.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:46:54.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:54.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:46:54.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:46:54.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:46:54.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:46:54.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 104 = '' 2026-02-20T22:46:54.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:46:54.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.status 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2026-02-20T22:46:55.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:46:55.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:46:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=104 2026-02-20T22:46:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 104 = 104 2026-02-20T22:46:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:46:55.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:56.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_10': 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10dabbb32bcc 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10dabbb32bcc 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:28 2026 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:28 2026 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:28 2026 2026-02-20T22:46:56.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:46:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:46:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:46:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 21a4a0c9-3c5e-4e1d-a979-7e282caacbc3 2026-02-20T22:46:56.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_10 snap 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.export 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.export 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.export 2026-02-20T22:46:56.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.export 2026-02-20T22:46:57.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:57.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.export 2026-02-20T22:46:57.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:46:57.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.export 2026-02-20T22:46:57.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_10.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_10.export 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_11 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_11 snap 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:46:57.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:46:58.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:58.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:46:59.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:46:59.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:46:59.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:46:59.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:46:59.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:46:59.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:46:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:59.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:46:59.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2026-02-20T22:46:59.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2026-02-20T22:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:46:59.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:46:59.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:46:59.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:46:59.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:46:59.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:59.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:59.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:46:59.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2026-02-20T22:46:59.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:46:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:46:59.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_11 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:46:59.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:46:59.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2026-02-20T22:46:59.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:46:59.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:46:59.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:46:59.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2026-02-20T22:47:00.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 106 2026-02-20T22:47:00.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_11 2026-02-20T22:47:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:00.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_11.status 2026-02-20T22:47:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_11.status 2026-02-20T22:47:00.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:00.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.status 2026-02-20T22:47:00.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_11.status 2026-02-20T22:47:00.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_11.status 2026-02-20T22:47:00.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.status 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.status 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2026-02-20T22:47:00.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:00.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:00.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=106 2026-02-20T22:47:00.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:00.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:00.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2026-02-20T22:47:01.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:01.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=45 2026-02-20T22:47:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 106 = 45 2026-02-20T22:47:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:01.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:01.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:01.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:01.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:01.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:01.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2026-02-20T22:47:01.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:01.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:01.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=45 2026-02-20T22:47:01.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 106 = 45 2026-02-20T22:47:01.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:01.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2026-02-20T22:47:02.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:02.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:02.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:02.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:02.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:02.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:02.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:02.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 106 = '' 2026-02-20T22:47:02.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:02.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:03.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.status 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2026-02-20T22:47:03.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:03.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=106 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 106 = 106 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:04.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2026-02-20T22:47:04.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_11': 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12134882b8c3 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12134882b8c3 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:42 2026 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:42 2026 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:42 2026 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: c1afeba9-ed53-4844-9dd9-34f5c5ee4c22 2026-02-20T22:47:04.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_11 snap 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.export 2026-02-20T22:47:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.export 2026-02-20T22:47:05.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.export 2026-02-20T22:47:05.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.export 2026-02-20T22:47:05.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:05.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.export 2026-02-20T22:47:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:05.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.export 2026-02-20T22:47:05.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_11.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_11.export 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_12 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_12 snap 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:05.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2026-02-20T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:07.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:07.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:07.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:07.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:07.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2026-02-20T22:47:07.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_12'] 2026-02-20T22:47:07.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:07.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:07.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_12 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:07.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:07.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:47:07.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:47:07.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:07.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:07.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:07.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_12 2026-02-20T22:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:07.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:07.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:07.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:07.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:07.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:07.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:07.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_12 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:07.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:07.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2026-02-20T22:47:08.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 108 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_12 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:08.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_12.status 2026-02-20T22:47:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_12.status 2026-02-20T22:47:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:08.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.status 2026-02-20T22:47:08.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_12.status 2026-02-20T22:47:08.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_12.status 2026-02-20T22:47:08.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:08.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.status 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.status 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2026-02-20T22:47:08.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:08.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=108 2026-02-20T22:47:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:08.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2026-02-20T22:47:09.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:09.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=46 2026-02-20T22:47:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 108 = 46 2026-02-20T22:47:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:09.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:09.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:09.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:09.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:09.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:09.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2026-02-20T22:47:09.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:09.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:09.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=46 2026-02-20T22:47:09.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 108 = 46 2026-02-20T22:47:09.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:09.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:10.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2026-02-20T22:47:10.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:10.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:10.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:10.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:10.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:10.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:10.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 108 = '' 2026-02-20T22:47:10.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:10.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.status 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2026-02-20T22:47:11.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:11.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=108 2026-02-20T22:47:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 108 = 108 2026-02-20T22:47:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2026-02-20T22:47:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:11.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:12.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2026-02-20T22:47:13.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_12': 2026-02-20T22:47:13.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:13.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12138a2cddc7 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12138a2cddc7 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:48 2026 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:48 2026 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:48 2026 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 97e62ce9-2919-43d0-b90f-ef29433498b6 2026-02-20T22:47:13.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:13.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:13.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_12 snap 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.export 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.export 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.export 2026-02-20T22:47:13.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.export 2026-02-20T22:47:13.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:13.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.export 2026-02-20T22:47:13.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:13.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.export 2026-02-20T22:47:13.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_12.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_12.export 2026-02-20T22:47:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_13 2026-02-20T22:47:13.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_13 snap 2026-02-20T22:47:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:13.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:14.132 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.6603189160643849) > weight (0.25) 2026-02-20T22:47:14.132 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8723561288517087) > weight (0.25) 2026-02-20T22:47:14.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.30185999188236434) > weight (0.25) 2026-02-20T22:47:14.133 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9123509728389806) > weight (0.25) 2026-02-20T22:47:14.133 INFO:tasks.rbd_mirror_thrash:waiting for 102.07488298135914 secs before thrashing 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2026-02-20T22:47:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:14.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:15.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2026-02-20T22:47:15.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:15.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:15.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:15.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:15.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:15.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:15.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2026-02-20T22:47:16.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:16.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:16.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:16.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_13 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:16.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:16.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2026-02-20T22:47:16.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 110 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_13 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:16.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_13.status 2026-02-20T22:47:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_13.status 2026-02-20T22:47:16.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:16.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.status 2026-02-20T22:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_13.status 2026-02-20T22:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_13.status 2026-02-20T22:47:16.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.status 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.status 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2026-02-20T22:47:16.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:16.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:16.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=110 2026-02-20T22:47:16.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:16.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:16.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2026-02-20T22:47:17.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:17.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=47 2026-02-20T22:47:17.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 110 = 47 2026-02-20T22:47:17.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:17.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:17.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2026-02-20T22:47:17.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:17.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=47 2026-02-20T22:47:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 110 = 47 2026-02-20T22:47:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:17.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:18.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:18.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:18.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2026-02-20T22:47:18.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:18.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:18.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:18.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:18.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:18.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:18.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:18.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 110 = '' 2026-02-20T22:47:18.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:18.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:20.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:20.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:20.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:20.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:20.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.status 2026-02-20T22:47:20.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2026-02-20T22:47:20.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:20.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=110 2026-02-20T22:47:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 110 = 110 2026-02-20T22:47:20.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:20.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:21.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2026-02-20T22:47:21.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_13': 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 125310c916cb 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.125310c916cb 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:21.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:37:57 2026 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:37:57 2026 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:37:57 2026 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 63774277-8720-40a9-a26f-36b0c3afe1c0 2026-02-20T22:47:21.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_13 snap 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.export 2026-02-20T22:47:21.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.export 2026-02-20T22:47:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.export 2026-02-20T22:47:21.143 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.export 2026-02-20T22:47:21.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.export 2026-02-20T22:47:21.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:21.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.export 2026-02-20T22:47:21.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_13.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_13.export 2026-02-20T22:47:21.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_14 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_14 snap 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:21.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2026-02-20T22:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:22.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:22.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:22.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:23.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:23.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:23.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:23.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:23.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:23.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:23.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:23.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:23.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:23.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:23.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2026-02-20T22:47:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:24.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:24.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:24.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:24.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_14 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:24.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2026-02-20T22:47:24.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 112 2026-02-20T22:47:24.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_14 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:24.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_14.status 2026-02-20T22:47:24.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_14.status 2026-02-20T22:47:24.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:24.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.status 2026-02-20T22:47:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_14.status 2026-02-20T22:47:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_14.status 2026-02-20T22:47:24.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:24.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.status 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.status 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2026-02-20T22:47:24.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:24.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:24.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=112 2026-02-20T22:47:24.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:24.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:25.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2026-02-20T22:47:25.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:25.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=48 2026-02-20T22:47:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 112 = 48 2026-02-20T22:47:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:25.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:25.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:25.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:25.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2026-02-20T22:47:25.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:25.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=48 2026-02-20T22:47:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 112 = 48 2026-02-20T22:47:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:25.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:26.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:26.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2026-02-20T22:47:26.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:26.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:26.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 112 = '' 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:26.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.status 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2026-02-20T22:47:28.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:28.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=112 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 112 = 112 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:28.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:29.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2026-02-20T22:47:29.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_14': 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253ce18bc29 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253ce18bc29 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:29.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:12 2026 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:12 2026 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:12 2026 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1ca8b7ab-8952-4355-92a0-9028c13306a9 2026-02-20T22:47:29.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_14 snap 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.export 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.export 2026-02-20T22:47:29.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.export 2026-02-20T22:47:29.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.export 2026-02-20T22:47:29.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:29.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.export 2026-02-20T22:47:29.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:29.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.export 2026-02-20T22:47:29.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_14.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_14.export 2026-02-20T22:47:29.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_15 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_15 snap 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:30.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:30.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:30.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:30.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:30.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:30.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2026-02-20T22:47:31.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:31.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:31.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:31.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:31.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:31.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:31.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:31.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:31.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:31.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2026-02-20T22:47:32.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2026-02-20T22:47:32.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:32.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:47:32.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:47:32.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:47:32.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:32.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:32.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:32.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2026-02-20T22:47:32.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2026-02-20T22:47:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:32.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:32.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:47:32.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:47:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2026-02-20T22:47:32.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:32.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:32.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:32.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:32.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_15 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:32.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2026-02-20T22:47:32.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 114 2026-02-20T22:47:32.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_15 2026-02-20T22:47:32.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_15.status 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_15.status 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.status 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_15.status 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_15.status 2026-02-20T22:47:32.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.status 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.status 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2026-02-20T22:47:32.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:32.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:32.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=114 2026-02-20T22:47:32.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:32.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:32.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:33.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2026-02-20T22:47:33.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:33.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=49 2026-02-20T22:47:33.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 114 = 49 2026-02-20T22:47:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:33.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:33.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2026-02-20T22:47:33.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:33.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=49 2026-02-20T22:47:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 114 = 49 2026-02-20T22:47:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:34.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2026-02-20T22:47:34.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:34.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:34.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:34.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:34.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:34.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:34.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:34.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 114 = '' 2026-02-20T22:47:34.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:34.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.status 2026-02-20T22:47:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2026-02-20T22:47:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:36.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=114 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 114 = 114 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:36.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:37.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_15': 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253d27cb2ef 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253d27cb2ef 2026-02-20T22:47:37.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:27 2026 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:27 2026 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:27 2026 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7d4c2418-4884-49bd-8deb-da886a79291a 2026-02-20T22:47:37.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_15 snap 2026-02-20T22:47:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:47:37.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.export 2026-02-20T22:47:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.export 2026-02-20T22:47:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.export 2026-02-20T22:47:37.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.export 2026-02-20T22:47:37.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:37.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.export 2026-02-20T22:47:37.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:37.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.export 2026-02-20T22:47:37.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_15.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_15.export 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_16 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_16 snap 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:37.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:38.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2026-02-20T22:47:38.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:38.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:39.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2026-02-20T22:47:39.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:39.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:39.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2026-02-20T22:47:39.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:39.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:39.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:39.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:39.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2026-02-20T22:47:40.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-20T22:47:40.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:40.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2026-02-20T22:47:40.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:40.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:40.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:47:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:47:40.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:47:40.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:40.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:40.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:40.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2026-02-20T22:47:40.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2026-02-20T22:47:40.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:40.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_16 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:40.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:40.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:47:40.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:47:40.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:40.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:40.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:40.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_16 2026-02-20T22:47:40.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:40.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:40.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:40.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:40.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_16 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:40.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2026-02-20T22:47:40.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 116 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_16 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:40.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:40.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_16.status 2026-02-20T22:47:40.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_16.status 2026-02-20T22:47:40.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:40.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.status 2026-02-20T22:47:40.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_16.status 2026-02-20T22:47:40.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_16.status 2026-02-20T22:47:40.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.status 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.status 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2026-02-20T22:47:40.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:41.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=116 2026-02-20T22:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:41.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2026-02-20T22:47:41.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:41.241 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=50 2026-02-20T22:47:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 116 = 50 2026-02-20T22:47:41.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:41.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:41.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2026-02-20T22:47:41.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:41.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=50 2026-02-20T22:47:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 116 = 50 2026-02-20T22:47:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:41.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:42.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2026-02-20T22:47:42.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:42.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:42.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 116 = '' 2026-02-20T22:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:42.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:47:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.status 2026-02-20T22:47:44.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2026-02-20T22:47:44.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:44.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=116 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 116 = 116 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:44.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:45.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_16': 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12537396c900 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12537396c900 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:33 2026 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:33 2026 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:33 2026 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: dd77e808-55a0-4c1a-a3d7-263a1ff1cd0d 2026-02-20T22:47:45.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_16 snap 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.export 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.export 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.export 2026-02-20T22:47:45.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.export 2026-02-20T22:47:45.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:45.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.export 2026-02-20T22:47:45.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:45.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.export 2026-02-20T22:47:45.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_16.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_16.export 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_17 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_17 snap 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:45.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:45.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:46.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:46.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:47.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:47.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:47.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:47.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:47.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:47.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:47.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:47.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:47.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2026-02-20T22:47:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-20T22:47:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:47.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_17 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:47.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:47.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:47:47.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:47:47.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_17 2026-02-20T22:47:47.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-20T22:47:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:47.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_17 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:47.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:47:47.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:47:47.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:47:47.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:47.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:47.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:47:47.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_17 2026-02-20T22:47:47.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_17'] 2026-02-20T22:47:47.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:47:47.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_17 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:47.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:47.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:47:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:47:47.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:47:47.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:47:47.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:47:47.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_17 2026-02-20T22:47:48.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:48.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_17 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2026-02-20T22:47:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:48.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2026-02-20T22:47:48.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 118 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_17 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:48.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:48.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_17.status 2026-02-20T22:47:48.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_17.status 2026-02-20T22:47:48.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:48.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.status 2026-02-20T22:47:48.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_17.status 2026-02-20T22:47:48.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_17.status 2026-02-20T22:47:48.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.status 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.status 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2026-02-20T22:47:48.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:48.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:48.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=118 2026-02-20T22:47:48.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:48.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:48.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:48.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2026-02-20T22:47:48.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:48.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:48.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:48.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 118 = '' 2026-02-20T22:47:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:48.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2026-02-20T22:47:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:49.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:49.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:49.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:49.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:49.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 118 = '' 2026-02-20T22:47:49.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:49.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:50.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:50.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:50.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:50.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:50.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.status 2026-02-20T22:47:50.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2026-02-20T22:47:50.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:50.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=118 2026-02-20T22:47:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 118 = 118 2026-02-20T22:47:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2026-02-20T22:47:50.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:50.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:51.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2026-02-20T22:47:51.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_17': 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10dad82f7fd3 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10dad82f7fd3 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:51.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:42 2026 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:42 2026 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:42 2026 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 58103425-2768-48ad-acca-e3c306a52862 2026-02-20T22:47:51.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_17 snap 2026-02-20T22:47:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:47:51.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.export 2026-02-20T22:47:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.export 2026-02-20T22:47:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.export 2026-02-20T22:47:51.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.export 2026-02-20T22:47:51.416 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7048765304889938) > weight (0.25) 2026-02-20T22:47:51.416 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-20T22:47:51.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-20T22:47:51.417 INFO:tasks.rbd_mirror_thrash:waiting for 2.6714853252693525 secs before reviving daemons 2026-02-20T22:47:51.417 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:47:51.410+0000 7f0025540640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 32270) UID: 1000 2026-02-20T22:47:51.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:51.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.export 2026-02-20T22:47:51.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:51.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.export 2026-02-20T22:47:51.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_17.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_17.export 2026-02-20T22:47:51.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:51.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:51.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_18 2026-02-20T22:47:51.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_18 snap 2026-02-20T22:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:51.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2026-02-20T22:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:53.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:53.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:54.089 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-20T22:47:54.089 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:47:54.090 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-20T22:47:54.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-20T22:47:54.090 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-20T22:47:54.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-20T22:47:54.092 INFO:tasks.rbd_mirror_thrash:waiting for 117.86851757862743 secs before thrashing 2026-02-20T22:47:54.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2026-02-20T22:47:54.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:54.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:54.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:47:54.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:47:54.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:47:54.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:54.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:47:54.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:47:54.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:47:54.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:47:54.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2026-02-20T22:47:54.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:47:54.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:47:54.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:47:54.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:47:54.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:47:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:47:54.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_18 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:54.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:54.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2026-02-20T22:47:55.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 120 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_18 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:55.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_18.status 2026-02-20T22:47:55.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_18.status 2026-02-20T22:47:55.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:55.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.status 2026-02-20T22:47:55.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_18.status 2026-02-20T22:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_18.status 2026-02-20T22:47:55.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.status 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.status 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2026-02-20T22:47:55.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:55.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:47:55.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=120 2026-02-20T22:47:55.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:55.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:55.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:47:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:55.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2026-02-20T22:47:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:55.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:55.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:55.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:55.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:55.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:55.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:55.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 120 = '' 2026-02-20T22:47:55.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:55.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2026-02-20T22:47:56.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:56.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:47:56.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:56.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:47:56.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:47:56.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:47:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:47:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 120 = '' 2026-02-20T22:47:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:47:56.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:47:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.status 2026-02-20T22:47:57.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2026-02-20T22:47:57.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:47:57.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:47:57.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=120 2026-02-20T22:47:57.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 120 = 120 2026-02-20T22:47:57.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:57.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:47:57.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:47:58.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2026-02-20T22:47:58.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_18': 2026-02-20T22:47:58.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:47:58.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213ade47684 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213ade47684 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:38:53 2026 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:38:53 2026 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:38:53 2026 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2e7d6018-b8f8-4aa9-a620-9abcba9ad630 2026-02-20T22:47:58.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_18 snap 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.export 2026-02-20T22:47:58.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.export 2026-02-20T22:47:58.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.export 2026-02-20T22:47:58.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.export 2026-02-20T22:47:58.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:58.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.export 2026-02-20T22:47:58.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:47:58.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.export 2026-02-20T22:47:58.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_18.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_18.export 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_19 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_19 snap 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:47:58.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:47:58.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:47:58.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:00.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:00.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:01.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:01.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2026-02-20T22:48:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:01.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:01.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:01.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:01.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:01.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:01.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:01.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2026-02-20T22:48:01.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_19'] 2026-02-20T22:48:01.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:01.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_19 2026-02-20T22:48:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:01.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:01.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:48:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:48:01.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:48:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:01.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_19 2026-02-20T22:48:01.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:01.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:01.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:01.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_19 2026-02-20T22:48:01.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:01.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2026-02-20T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:01.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2026-02-20T22:48:02.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 122 2026-02-20T22:48:02.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_19 2026-02-20T22:48:02.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:02.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:02.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:02.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:02.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:02.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_19.status 2026-02-20T22:48:02.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_19.status 2026-02-20T22:48:02.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:02.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.status 2026-02-20T22:48:02.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_19.status 2026-02-20T22:48:02.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_19.status 2026-02-20T22:48:02.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:02.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:02.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.status 2026-02-20T22:48:02.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:02.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:02.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:02.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.status 2026-02-20T22:48:02.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2026-02-20T22:48:02.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:02.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:02.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=122 2026-02-20T22:48:02.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:02.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:02.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2026-02-20T22:48:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:03.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:03.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 122 = '' 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:03.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:03.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2026-02-20T22:48:03.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:03.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:03.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:03.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 122 = '' 2026-02-20T22:48:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:03.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:04.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.status 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2026-02-20T22:48:04.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:04.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=122 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 122 = 122 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:04.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:04.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:05.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_19': 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 124981ae5b5e 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.124981ae5b5e 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:00 2026 2026-02-20T22:48:05.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:00 2026 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:00 2026 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3c21132a-e262-46d0-a004-e6e7cbc448c9 2026-02-20T22:48:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_19 snap 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.export 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.export 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.export 2026-02-20T22:48:05.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.export 2026-02-20T22:48:05.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:05.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.export 2026-02-20T22:48:05.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:05.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.export 2026-02-20T22:48:05.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_19.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_19.export 2026-02-20T22:48:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_20 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_20 snap 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:08.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:08.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:08.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:08.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:08.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:08.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:08.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:08.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:08.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:08.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:08.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:08.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:08.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2026-02-20T22:48:08.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-20T22:48:08.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:08.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2026-02-20T22:48:08.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:08.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:48:08.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:48:08.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:08.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:08.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:08.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2026-02-20T22:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2026-02-20T22:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:08.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_20 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:48:08.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:48:08.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:48:08.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:48:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:48:08.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:08.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:08.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:08.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_20 2026-02-20T22:48:08.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:08.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:08.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:08.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_20 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:08.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2026-02-20T22:48:09.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 124 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_20 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:09.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_20.status 2026-02-20T22:48:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_20.status 2026-02-20T22:48:09.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.status 2026-02-20T22:48:09.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_20.status 2026-02-20T22:48:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_20.status 2026-02-20T22:48:09.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:09.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:09.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.status 2026-02-20T22:48:09.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.status 2026-02-20T22:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2026-02-20T22:48:09.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:09.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=124 2026-02-20T22:48:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:09.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2026-02-20T22:48:09.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:10.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:10.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:10.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:10.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:10.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:10.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 124 = '' 2026-02-20T22:48:10.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:10.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2026-02-20T22:48:10.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:10.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 124 = '' 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:10.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:11.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.status 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2026-02-20T22:48:11.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:11.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:11.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=124 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 124 = 124 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:11.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:12.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_20': 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253bb13ad62 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253bb13ad62 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:15 2026 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:15 2026 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:15 2026 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 84a30fa8-259c-4c04-88f1-8a4e44ec7139 2026-02-20T22:48:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_20 snap 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:12.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:12.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.export 2026-02-20T22:48:12.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.export 2026-02-20T22:48:12.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.export 2026-02-20T22:48:12.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.export 2026-02-20T22:48:12.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:12.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.export 2026-02-20T22:48:12.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:12.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.export 2026-02-20T22:48:12.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_20.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_20.export 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_21 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_21 snap 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:12.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:12.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:14.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:14.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:15.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:15.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:15.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:15.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:15.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:15.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:15.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:15.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:15.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:15.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2026-02-20T22:48:15.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:15.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_21 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:15.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2026-02-20T22:48:15.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:15.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:15.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:15.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2026-02-20T22:48:16.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 126 2026-02-20T22:48:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_21 2026-02-20T22:48:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:16.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:16.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:16.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_21.status 2026-02-20T22:48:16.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_21.status 2026-02-20T22:48:16.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:16.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.status 2026-02-20T22:48:16.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_21.status 2026-02-20T22:48:16.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_21.status 2026-02-20T22:48:16.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:16.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:16.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.status 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.status 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2026-02-20T22:48:16.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:16.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=126 2026-02-20T22:48:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:16.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:16.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:16.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:16.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2026-02-20T22:48:16.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:17.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:17.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=55 2026-02-20T22:48:17.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 126 = 55 2026-02-20T22:48:17.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:17.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2026-02-20T22:48:17.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:17.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:17.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=55 2026-02-20T22:48:17.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 126 = 55 2026-02-20T22:48:17.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:17.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:18.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2026-02-20T22:48:18.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:18.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:18.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 126 = '' 2026-02-20T22:48:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:19.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.status 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2026-02-20T22:48:19.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=126 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 126 = 126 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:19.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:20.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2026-02-20T22:48:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_21': 2026-02-20T22:48:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 10da547d8080 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.10da547d8080 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:22 2026 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:22 2026 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:22 2026 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:20.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 69840726-9864-4961-8865-3713146736b0 2026-02-20T22:48:20.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:20.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_21 snap 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.export 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.export 2026-02-20T22:48:20.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.export 2026-02-20T22:48:20.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.export 2026-02-20T22:48:21.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:21.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.export 2026-02-20T22:48:21.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.export 2026-02-20T22:48:21.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_21.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_21.export 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_22 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_22 snap 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:21.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2026-02-20T22:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:22.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:22.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:23.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:23.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2026-02-20T22:48:23.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:23.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:23.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:23.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:23.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:23.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2026-02-20T22:48:23.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:23.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_22 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:23.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2026-02-20T22:48:24.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 128 2026-02-20T22:48:24.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_22 2026-02-20T22:48:24.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:24.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_22.status 2026-02-20T22:48:24.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_22.status 2026-02-20T22:48:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:24.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.status 2026-02-20T22:48:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_22.status 2026-02-20T22:48:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_22.status 2026-02-20T22:48:24.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.status 2026-02-20T22:48:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:24.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.status 2026-02-20T22:48:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2026-02-20T22:48:24.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:25.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:25.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=128 2026-02-20T22:48:25.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:25.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:25.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:25.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:25.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:25.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2026-02-20T22:48:25.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:25.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:25.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=56 2026-02-20T22:48:25.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 128 = 56 2026-02-20T22:48:25.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:25.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:25.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2026-02-20T22:48:25.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:25.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=56 2026-02-20T22:48:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 128 = 56 2026-02-20T22:48:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:25.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:26.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2026-02-20T22:48:26.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:26.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:26.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:26.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 128 = '' 2026-02-20T22:48:26.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:26.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:28.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:28.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.status 2026-02-20T22:48:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2026-02-20T22:48:28.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:28.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=128 2026-02-20T22:48:28.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 128 = 128 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:28.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:29.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2026-02-20T22:48:29.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_22': 2026-02-20T22:48:29.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213e75cf49f 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213e75cf49f 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:36 2026 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:36 2026 2026-02-20T22:48:29.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:36 2026 2026-02-20T22:48:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 49060f6a-cc8c-4444-ac78-938dac73f064 2026-02-20T22:48:29.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:29.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_22 snap 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.export 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.export 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.export 2026-02-20T22:48:29.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.export 2026-02-20T22:48:29.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:29.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.export 2026-02-20T22:48:29.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:29.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.export 2026-02-20T22:48:29.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_22.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_22.export 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_23 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_23 snap 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:29.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:30.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:31.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2026-02-20T22:48:31.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:31.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:31.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:31.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:31.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:31.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:31.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:31.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:31.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:31.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:31.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:31.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:31.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:31.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:31.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2026-02-20T22:48:31.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2026-02-20T22:48:31.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:31.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_23 2026-02-20T22:48:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:48:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:31.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:48:31.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:31.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:31.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:31.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:48:31.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:48:31.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:48:31.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:31.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_23 2026-02-20T22:48:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2026-02-20T22:48:31.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:31.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:31.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:31.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_23 2026-02-20T22:48:31.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:48:31.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:31.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:31.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:31.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:48:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:48:31.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:48:31.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:31.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:31.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:31.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_23 2026-02-20T22:48:31.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_23'] 2026-02-20T22:48:31.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:31.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_23 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:48:31.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:31.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:48:31.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:48:31.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:31.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:31.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:48:31.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:48:31.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:48:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:31.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_23 2026-02-20T22:48:32.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:32.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:32.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:32.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:32.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_23 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:32.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:32.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2026-02-20T22:48:32.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 130 2026-02-20T22:48:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_23 2026-02-20T22:48:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:32.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_23.status 2026-02-20T22:48:32.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_23.status 2026-02-20T22:48:32.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:32.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.status 2026-02-20T22:48:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_23.status 2026-02-20T22:48:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_23.status 2026-02-20T22:48:32.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:32.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:32.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.status 2026-02-20T22:48:32.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:32.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:32.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:32.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.status 2026-02-20T22:48:32.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2026-02-20T22:48:32.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:32.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:32.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=130 2026-02-20T22:48:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:32.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:33.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2026-02-20T22:48:33.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:33.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=57 2026-02-20T22:48:33.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 130 = 57 2026-02-20T22:48:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:33.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:33.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:33.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:33.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:33.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:33.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:33.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2026-02-20T22:48:33.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:33.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:33.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=57 2026-02-20T22:48:33.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 130 = 57 2026-02-20T22:48:33.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:33.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:34.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:34.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2026-02-20T22:48:34.297 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:34.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:34.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:34.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:34.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:34.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 130 = '' 2026-02-20T22:48:34.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:34.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:35.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.status 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2026-02-20T22:48:35.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:35.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=130 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 130 = 130 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:36.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_23': 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12494eb0082e 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12494eb0082e 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:37.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:50 2026 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:50 2026 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:50 2026 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 357d90ab-5f43-4a44-b85f-74b7bbc1d809 2026-02-20T22:48:37.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_23 snap 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:37.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:37.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.export 2026-02-20T22:48:37.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.export 2026-02-20T22:48:37.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.export 2026-02-20T22:48:37.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.export 2026-02-20T22:48:37.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:37.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.export 2026-02-20T22:48:37.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:37.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.export 2026-02-20T22:48:37.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_23.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_23.export 2026-02-20T22:48:37.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_24 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_24 snap 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:37.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:38.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2026-02-20T22:48:38.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:39.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2026-02-20T22:48:39.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:39.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2026-02-20T22:48:39.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:39.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:39.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:39.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:39.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:39.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:39.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:39.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:39.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:39.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2026-02-20T22:48:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-20T22:48:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:39.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:39.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:39.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_24 2026-02-20T22:48:39.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:39.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:39.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:48:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:48:39.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:48:39.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:39.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:39.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:39.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_24 2026-02-20T22:48:39.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_24'] 2026-02-20T22:48:39.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:39.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_24 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:48:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:48:39.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:39.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:39.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:39.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_24 2026-02-20T22:48:40.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:40.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:40.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:40.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:40.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:40.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_24 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:40.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2026-02-20T22:48:40.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 132 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_24 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:40.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_24.status 2026-02-20T22:48:40.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_24.status 2026-02-20T22:48:40.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:40.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.status 2026-02-20T22:48:40.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_24.status 2026-02-20T22:48:40.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_24.status 2026-02-20T22:48:40.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.status 2026-02-20T22:48:40.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.status 2026-02-20T22:48:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2026-02-20T22:48:40.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:40.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:40.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=132 2026-02-20T22:48:40.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:40.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:40.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:41.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2026-02-20T22:48:41.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:41.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=58 2026-02-20T22:48:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 132 = 58 2026-02-20T22:48:41.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:41.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:41.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:41.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2026-02-20T22:48:41.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:41.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:41.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=58 2026-02-20T22:48:41.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 132 = 58 2026-02-20T22:48:41.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:41.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:42.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2026-02-20T22:48:42.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:42.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:42.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:42.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:42.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:42.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:42.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:42.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 132 = '' 2026-02-20T22:48:42.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:42.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.status 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2026-02-20T22:48:43.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:43.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=132 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 132 = 132 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:43.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:44.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2026-02-20T22:48:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_24': 2026-02-20T22:48:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253ce151603 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253ce151603 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:39:57 2026 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:39:57 2026 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:39:57 2026 2026-02-20T22:48:45.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: eaa9a84f-e88b-4266-b522-69fac662b35d 2026-02-20T22:48:45.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_24 snap 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:45.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.export 2026-02-20T22:48:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.export 2026-02-20T22:48:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.export 2026-02-20T22:48:45.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.export 2026-02-20T22:48:45.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:45.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.export 2026-02-20T22:48:45.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:45.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.export 2026-02-20T22:48:45.418 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_24.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_24.export 2026-02-20T22:48:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:45.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_25 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_25 snap 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:45.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:46.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:46.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:46.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:46.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:46.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:46.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2026-02-20T22:48:47.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:47.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:47.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:47.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:47.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:47.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:47.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:47.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:47.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:47.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:47.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2026-02-20T22:48:47.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:47.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:47.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:47.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:47.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_25 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:47.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:47.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2026-02-20T22:48:48.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 134 2026-02-20T22:48:48.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_25 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_25.status 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_25.status 2026-02-20T22:48:48.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:48.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.status 2026-02-20T22:48:48.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_25.status 2026-02-20T22:48:48.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_25.status 2026-02-20T22:48:48.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:48.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:48.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.status 2026-02-20T22:48:48.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:48.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.status 2026-02-20T22:48:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2026-02-20T22:48:48.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:48.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=134 2026-02-20T22:48:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:48.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:49.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2026-02-20T22:48:49.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:49.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=59 2026-02-20T22:48:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 134 = 59 2026-02-20T22:48:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:49.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:49.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2026-02-20T22:48:49.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:49.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=59 2026-02-20T22:48:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 134 = 59 2026-02-20T22:48:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:49.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:50.335 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2026-02-20T22:48:50.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:50.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:50.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:50.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 134 = '' 2026-02-20T22:48:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:50.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:51.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.status 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2026-02-20T22:48:51.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:52.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=134 2026-02-20T22:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 134 = 134 2026-02-20T22:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2026-02-20T22:48:52.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:48:52.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:53.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2026-02-20T22:48:53.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_25': 2026-02-20T22:48:53.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:48:53.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:48:53.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1262e0bbf035 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1262e0bbf035 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:08 2026 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:08 2026 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:08 2026 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 07156c03-24cc-4962-b7eb-f044afb9ccd1 2026-02-20T22:48:53.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_25 snap 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.export 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.export 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.export 2026-02-20T22:48:53.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.export 2026-02-20T22:48:53.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:53.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.export 2026-02-20T22:48:53.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:48:53.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.export 2026-02-20T22:48:53.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_25.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_25.export 2026-02-20T22:48:53.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:53.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:48:53.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_26 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_26 snap 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:48:53.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:48:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2026-02-20T22:48:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:54.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:48:55.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2026-02-20T22:48:55.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:55.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:55.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:55.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:48:55.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:48:55.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:48:55.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:55.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:48:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:48:55.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:48:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:48:55.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2026-02-20T22:48:55.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-20T22:48:55.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:55.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2026-02-20T22:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:55.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:55.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:55.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:55.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:48:55.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:48:55.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:48:55.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:55.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:55.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:48:55.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2026-02-20T22:48:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-20T22:48:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:55.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:55.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:48:55.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:48:55.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:48:55.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:55.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:55.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:48:55.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2026-02-20T22:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2026-02-20T22:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:48:56.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_26 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:48:56.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:48:56.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:48:56.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:48:56.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:48:56.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:48:56.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:56.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:56.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:48:56.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_26 2026-02-20T22:48:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:48:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:48:56.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:48:56.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:48:56.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:48:56.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:48:56.107 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_26 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2026-02-20T22:48:56.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:56.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:56.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:56.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2026-02-20T22:48:56.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8140983722723197) > weight (0.25) 2026-02-20T22:48:56.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7400782499732622) > weight (0.25) 2026-02-20T22:48:56.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.34339404257903505) > weight (0.25) 2026-02-20T22:48:56.209 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.6729511880140164) > weight (0.25) 2026-02-20T22:48:56.209 INFO:tasks.rbd_mirror_thrash:waiting for 76.607682053504 secs before thrashing 2026-02-20T22:48:56.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 136 2026-02-20T22:48:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_26 2026-02-20T22:48:56.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_26.status 2026-02-20T22:48:56.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_26.status 2026-02-20T22:48:56.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:56.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.status 2026-02-20T22:48:56.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_26.status 2026-02-20T22:48:56.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_26.status 2026-02-20T22:48:56.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:48:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.status 2026-02-20T22:48:56.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:48:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.status 2026-02-20T22:48:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2026-02-20T22:48:56.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:56.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:48:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=136 2026-02-20T22:48:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:56.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2026-02-20T22:48:57.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:57.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:57.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=60 2026-02-20T22:48:57.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 136 = 60 2026-02-20T22:48:57.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:57.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:48:57.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:57.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2026-02-20T22:48:57.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:57.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=60 2026-02-20T22:48:57.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 136 = 60 2026-02-20T22:48:57.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:57.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:48:58.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:58.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:58.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:58.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:58.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:58.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2026-02-20T22:48:58.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:48:58.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:48:58.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:48:58.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:48:58.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:48:58.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:48:58.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:48:58.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 136 = '' 2026-02-20T22:48:58.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:48:58.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:48:59.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.status 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2026-02-20T22:48:59.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:00.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=136 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 136 = 136 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:01.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_26': 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1262cb23a61f 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1262cb23a61f 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:01.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:17 2026 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:17 2026 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:17 2026 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 96d0ea21-daaa-43c4-b57c-6d4a1854bd5a 2026-02-20T22:49:01.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_26 snap 2026-02-20T22:49:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:01.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.export 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.export 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.export 2026-02-20T22:49:01.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.export 2026-02-20T22:49:01.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:01.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.export 2026-02-20T22:49:01.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:01.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.export 2026-02-20T22:49:01.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_26.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_26.export 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_27 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_27 snap 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:01.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2026-02-20T22:49:02.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:02.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2026-02-20T22:49:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:03.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:03.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:03.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:03.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:03.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:03.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:03.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:03.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2026-02-20T22:49:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-20T22:49:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:03.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:03.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:03.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:03.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:03.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:03.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2026-02-20T22:49:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2026-02-20T22:49:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:03.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:03.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:03.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:49:03.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:49:03.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:03.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:03.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:03.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2026-02-20T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:04.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_27 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:04.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:04.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2026-02-20T22:49:04.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 138 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_27 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_27.status 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_27.status 2026-02-20T22:49:04.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.status 2026-02-20T22:49:04.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_27.status 2026-02-20T22:49:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_27.status 2026-02-20T22:49:04.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:04.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.status 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.status 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2026-02-20T22:49:04.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:04.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:04.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=138 2026-02-20T22:49:04.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:04.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:04.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:05.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2026-02-20T22:49:05.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:05.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:05.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=61 2026-02-20T22:49:05.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 138 = 61 2026-02-20T22:49:05.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:05.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:05.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2026-02-20T22:49:05.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:05.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=61 2026-02-20T22:49:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 138 = 61 2026-02-20T22:49:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:05.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2026-02-20T22:49:06.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:06.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:06.386 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 138 = '' 2026-02-20T22:49:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:06.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.status 2026-02-20T22:49:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2026-02-20T22:49:07.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=138 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 138 = 138 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:08.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:08.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:08.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:08.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:09.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_27': 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 125355de2e70 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.125355de2e70 2026-02-20T22:49:09.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:25 2026 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:25 2026 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:25 2026 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9dacbf76-f085-4405-b87d-fe7a44b68efb 2026-02-20T22:49:09.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_27 snap 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:09.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.export 2026-02-20T22:49:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.export 2026-02-20T22:49:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.export 2026-02-20T22:49:09.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.export 2026-02-20T22:49:09.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:09.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.export 2026-02-20T22:49:09.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:09.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.export 2026-02-20T22:49:09.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_27.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_27.export 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_28 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_28 snap 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:09.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2026-02-20T22:49:10.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:10.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:11.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2026-02-20T22:49:11.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:11.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:11.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:11.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:11.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:11.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:11.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2026-02-20T22:49:11.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:11.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:11.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:11.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:11.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:11.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:11.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2026-02-20T22:49:11.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:11.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:11.852 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:11.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:11.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:11.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:11.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2026-02-20T22:49:11.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-20T22:49:11.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:11.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:11.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:11.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:11.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:11.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:11.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:11.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:11.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:11.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2026-02-20T22:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-20T22:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:11.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2026-02-20T22:49:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:49:11.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:49:11.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:11.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:49:11.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:49:12.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:12.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2026-02-20T22:49:12.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2026-02-20T22:49:12.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:12.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:12.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_28 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:49:12.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:12.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:49:12.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:49:12.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:12.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:12.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:12.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_28 2026-02-20T22:49:12.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:12.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:12.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:12.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:12.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_28 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:12.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2026-02-20T22:49:12.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 140 2026-02-20T22:49:12.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_28 2026-02-20T22:49:12.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_28.status 2026-02-20T22:49:12.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_28.status 2026-02-20T22:49:12.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:12.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.status 2026-02-20T22:49:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_28.status 2026-02-20T22:49:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_28.status 2026-02-20T22:49:12.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.status 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.status 2026-02-20T22:49:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2026-02-20T22:49:12.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:12.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:12.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=140 2026-02-20T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:12.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:13.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2026-02-20T22:49:13.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:13.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=62 2026-02-20T22:49:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 140 = 62 2026-02-20T22:49:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:13.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2026-02-20T22:49:13.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:13.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:13.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=62 2026-02-20T22:49:13.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 140 = 62 2026-02-20T22:49:13.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:13.561 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2026-02-20T22:49:14.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:14.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:14.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:14.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:14.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:14.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:14.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:14.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 140 = '' 2026-02-20T22:49:14.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:14.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.status 2026-02-20T22:49:15.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2026-02-20T22:49:15.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:16.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=140 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 140 = 140 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:16.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:16.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:17.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2026-02-20T22:49:17.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_28': 2026-02-20T22:49:17.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:17.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1262e2bbfa8c 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1262e2bbfa8c 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:32 2026 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:32 2026 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:32 2026 2026-02-20T22:49:17.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:17.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:17.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:17.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: f3e3b67e-63fc-4f0c-a1ad-75bec4cf631a 2026-02-20T22:49:17.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_28 snap 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:17.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:17.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.export 2026-02-20T22:49:17.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.export 2026-02-20T22:49:17.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.export 2026-02-20T22:49:17.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.export 2026-02-20T22:49:17.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:17.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.export 2026-02-20T22:49:17.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:17.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.export 2026-02-20T22:49:17.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_28.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_28.export 2026-02-20T22:49:17.509 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_29 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_29 snap 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:17.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:17.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:18.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2026-02-20T22:49:18.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:18.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:18.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:18.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:19.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2026-02-20T22:49:19.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:19.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:19.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:19.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:19.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:19.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:19.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:19.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:19.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:19.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2026-02-20T22:49:19.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2026-02-20T22:49:19.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:19.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:19.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:19.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:19.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:20.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2026-02-20T22:49:20.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:20.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:20.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:20.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:20.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:20.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:20.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:20.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:20.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_29 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:20.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:20.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2026-02-20T22:49:20.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 142 2026-02-20T22:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_29 2026-02-20T22:49:20.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_29.status 2026-02-20T22:49:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_29.status 2026-02-20T22:49:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:20.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.status 2026-02-20T22:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_29.status 2026-02-20T22:49:20.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_29.status 2026-02-20T22:49:20.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.status 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:20.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.status 2026-02-20T22:49:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2026-02-20T22:49:20.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:20.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=142 2026-02-20T22:49:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:20.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:21.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2026-02-20T22:49:21.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:21.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=63 2026-02-20T22:49:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 142 = 63 2026-02-20T22:49:21.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:21.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:21.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:21.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:21.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:21.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:21.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:21.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2026-02-20T22:49:21.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:21.635 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=63 2026-02-20T22:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 142 = 63 2026-02-20T22:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:21.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:22.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2026-02-20T22:49:22.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:22.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:22.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:22.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 142 = '' 2026-02-20T22:49:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:22.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:24.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.status 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2026-02-20T22:49:24.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:24.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=142 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 142 = 142 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:24.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:24.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:25.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_29': 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213e2597314 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213e2597314 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:46 2026 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:46 2026 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:46 2026 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: b44f2e80-554c-46e7-9cff-22a5d35c7769 2026-02-20T22:49:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_29 snap 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.export 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.export 2026-02-20T22:49:25.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.export 2026-02-20T22:49:25.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.export 2026-02-20T22:49:25.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:25.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.export 2026-02-20T22:49:25.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:25.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.export 2026-02-20T22:49:25.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_29.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_29.export 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_30 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_30 snap 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:25.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:25.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:25.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:25.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2026-02-20T22:49:26.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:27.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:27.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:27.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:27.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:27.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:27.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2026-02-20T22:49:28.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2026-02-20T22:49:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:28.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:28.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:28.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:28.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:28.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:28.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:28.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:28.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:28.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:28.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2026-02-20T22:49:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:28.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:28.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:28.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_30 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:28.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2026-02-20T22:49:28.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 144 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_30 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:28.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_30.status 2026-02-20T22:49:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_30.status 2026-02-20T22:49:28.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.status 2026-02-20T22:49:28.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_30.status 2026-02-20T22:49:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_30.status 2026-02-20T22:49:28.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.status 2026-02-20T22:49:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:28.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.status 2026-02-20T22:49:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2026-02-20T22:49:28.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:28.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=144 2026-02-20T22:49:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:28.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:29.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2026-02-20T22:49:29.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:29.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=64 2026-02-20T22:49:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 144 = 64 2026-02-20T22:49:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:29.220 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2026-02-20T22:49:29.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:29.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:29.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=64 2026-02-20T22:49:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 144 = 64 2026-02-20T22:49:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:30.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2026-02-20T22:49:30.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:30.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:30.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:30.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:30.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 144 = '' 2026-02-20T22:49:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:30.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:32.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.status 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2026-02-20T22:49:32.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:32.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:32.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=144 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 144 = 144 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:32.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:33.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2026-02-20T22:49:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_30': 2026-02-20T22:49:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:33.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12496b78b36e 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12496b78b36e 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:40:54 2026 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:40:54 2026 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:40:54 2026 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:33.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 76b57b0a-af4f-4c2a-9a97-d9e658cc43f5 2026-02-20T22:49:33.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_30 snap 2026-02-20T22:49:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:49:33.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.export 2026-02-20T22:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.export 2026-02-20T22:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.export 2026-02-20T22:49:33.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.export 2026-02-20T22:49:33.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:33.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.export 2026-02-20T22:49:33.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:33.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.export 2026-02-20T22:49:33.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_30.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_30.export 2026-02-20T22:49:33.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:33.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_31 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_31 snap 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:33.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2026-02-20T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:34.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:34.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:35.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:35.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:35.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:35.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:35.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:35.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:35.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2026-02-20T22:49:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-20T22:49:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:36.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:36.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:36.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:36.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:36.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:36.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:36.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:36.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2026-02-20T22:49:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2026-02-20T22:49:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:36.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:49:36.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:49:36.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:49:36.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:36.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:36.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:36.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2026-02-20T22:49:36.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:36.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:36.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_31 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:36.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:36.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2026-02-20T22:49:36.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 146 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_31 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:36.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_31.status 2026-02-20T22:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_31.status 2026-02-20T22:49:36.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.status 2026-02-20T22:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_31.status 2026-02-20T22:49:36.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_31.status 2026-02-20T22:49:36.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.status 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.status 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2026-02-20T22:49:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:36.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=146 2026-02-20T22:49:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:36.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:37.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:37.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2026-02-20T22:49:37.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:37.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:37.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=65 2026-02-20T22:49:37.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 146 = 65 2026-02-20T22:49:37.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:37.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:37.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2026-02-20T22:49:37.845 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:37.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:37.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:37.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:37.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:37.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:37.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 146 = '' 2026-02-20T22:49:37.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:37.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2026-02-20T22:49:38.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:38.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:38.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:38.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:38.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:38.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 146 = '' 2026-02-20T22:49:38.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:38.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.status 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2026-02-20T22:49:40.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:40.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=146 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 146 = 146 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:40.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:40.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:40.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:41.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_31': 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12534357f564 2026-02-20T22:49:41.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12534357f564 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:02 2026 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:02 2026 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:02 2026 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:41.378 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 0d6df238-c085-4a6e-ad2d-7aa25e3b6e40 2026-02-20T22:49:41.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_31 snap 2026-02-20T22:49:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:49:41.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.export 2026-02-20T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.export 2026-02-20T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.export 2026-02-20T22:49:41.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.export 2026-02-20T22:49:41.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:41.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.export 2026-02-20T22:49:41.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:41.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.export 2026-02-20T22:49:41.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_31.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_31.export 2026-02-20T22:49:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_32 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_32 snap 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:42.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:42.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:42.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:42.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2026-02-20T22:49:43.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:43.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:43.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:43.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:43.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:43.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2026-02-20T22:49:43.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:43.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:43.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:43.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:43.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:43.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:43.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:43.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:43.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2026-02-20T22:49:44.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2026-02-20T22:49:44.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:44.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:44.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:44.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:44.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2026-02-20T22:49:44.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2026-02-20T22:49:44.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:44.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:49:44.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:44.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:49:44.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:49:44.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:49:44.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:44.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:44.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:44.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2026-02-20T22:49:44.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2026-02-20T22:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:44.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:44.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:49:44.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:49:44.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:49:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:49:44.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2026-02-20T22:49:44.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:44.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_32 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:44.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:44.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2026-02-20T22:49:44.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 148 2026-02-20T22:49:44.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_32 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:44.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_32.status 2026-02-20T22:49:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_32.status 2026-02-20T22:49:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:44.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.status 2026-02-20T22:49:44.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_32.status 2026-02-20T22:49:44.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_32.status 2026-02-20T22:49:44.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.status 2026-02-20T22:49:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:44.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.status 2026-02-20T22:49:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2026-02-20T22:49:44.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:45.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=148 2026-02-20T22:49:45.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:45.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:45.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:45.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2026-02-20T22:49:45.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:45.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:45.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=66 2026-02-20T22:49:45.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 148 = 66 2026-02-20T22:49:45.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:45.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2026-02-20T22:49:45.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:45.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=66 2026-02-20T22:49:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 148 = 66 2026-02-20T22:49:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:45.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:46.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2026-02-20T22:49:46.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:46.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:46.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:46.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:46.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 148 = '' 2026-02-20T22:49:46.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:46.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:49:48.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:48.130 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.status 2026-02-20T22:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2026-02-20T22:49:48.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:48.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:48.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=148 2026-02-20T22:49:48.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 148 = 148 2026-02-20T22:49:48.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:48.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:49.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_32': 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 126213a21649 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.126213a21649 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:49.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:08 2026 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:08 2026 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:08 2026 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3e048625-939b-485e-ae8c-d4b0fec7ae1d 2026-02-20T22:49:49.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_32 snap 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.export 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.export 2026-02-20T22:49:49.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.export 2026-02-20T22:49:49.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.export 2026-02-20T22:49:49.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:49.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.export 2026-02-20T22:49:49.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:49.516 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.export 2026-02-20T22:49:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_32.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_32.export 2026-02-20T22:49:49.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:49.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:49.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_33 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_33 snap 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:49.618 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:50.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2026-02-20T22:49:50.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:50.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:50.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2026-02-20T22:49:51.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:51.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2026-02-20T22:49:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:51.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:51.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:51.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:51.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:51.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:51.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:51.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:51.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:51.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2026-02-20T22:49:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2026-02-20T22:49:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:51.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:51.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:51.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:51.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:51.798 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:51.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:51.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2026-02-20T22:49:51.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:51.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:51.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:51.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:51.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:51.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_33 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:51.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2026-02-20T22:49:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:51.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2026-02-20T22:49:51.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.46041028975245757) > weight (0.25) 2026-02-20T22:49:51.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7411197308422619) > weight (0.25) 2026-02-20T22:49:51.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.345585052013219) > weight (0.25) 2026-02-20T22:49:51.962 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9393374383943475) > weight (0.25) 2026-02-20T22:49:51.962 INFO:tasks.rbd_mirror_thrash:waiting for 72.36327617783417 secs before thrashing 2026-02-20T22:49:52.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 150 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_33 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:52.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_33.status 2026-02-20T22:49:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_33.status 2026-02-20T22:49:52.729 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.status 2026-02-20T22:49:52.730 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_33.status 2026-02-20T22:49:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_33.status 2026-02-20T22:49:52.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:52.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.status 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.status 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2026-02-20T22:49:52.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:52.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:52.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=150 2026-02-20T22:49:52.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:52.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:52.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:52.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2026-02-20T22:49:52.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:52.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:52.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:53.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:53.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:53.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 150 = '' 2026-02-20T22:49:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:53.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2026-02-20T22:49:53.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:53.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:53.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:49:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:49:53.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:49:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:49:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 150 = '' 2026-02-20T22:49:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:53.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:49:54.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.status 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2026-02-20T22:49:54.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:54.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:49:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=150 2026-02-20T22:49:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 150 = 150 2026-02-20T22:49:54.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:49:54.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:55.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_33': 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213cd1c539e 2026-02-20T22:49:55.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213cd1c539e 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:14 2026 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:14 2026 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:14 2026 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:49:55.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 127d9d72-c25f-4d41-a45b-12cfcc97b19d 2026-02-20T22:49:55.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_33 snap 2026-02-20T22:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:49:55.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.export 2026-02-20T22:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.export 2026-02-20T22:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.export 2026-02-20T22:49:55.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.export 2026-02-20T22:49:55.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:55.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.export 2026-02-20T22:49:55.630 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:49:55.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.export 2026-02-20T22:49:55.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_33.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_33.export 2026-02-20T22:49:55.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_34 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_34 snap 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:49:55.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:57.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:49:57.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:49:57.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:57.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:49:58.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:49:58.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:49:58.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:49:58.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:49:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:58.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:58.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:49:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:49:58.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:49:58.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:49:58.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2026-02-20T22:49:58.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-20T22:49:58.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:58.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:49:58.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:49:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:49:58.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:58.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:58.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:49:58.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2026-02-20T22:49:58.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2026-02-20T22:49:58.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:49:58.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_34 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:49:58.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:49:58.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:49:58.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:58.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:58.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:49:58.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_34 2026-02-20T22:49:58.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:49:58.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:49:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:49:58.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:49:58.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_34 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2026-02-20T22:49:59.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 152 2026-02-20T22:49:59.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_34 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:59.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_34.status 2026-02-20T22:49:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_34.status 2026-02-20T22:49:59.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.status 2026-02-20T22:49:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_34.status 2026-02-20T22:49:59.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_34.status 2026-02-20T22:49:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:49:59.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:49:59.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.status 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.status 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2026-02-20T22:49:59.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:49:59.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:49:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=152 2026-02-20T22:49:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:49:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:49:59.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2026-02-20T22:49:59.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:00.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:00.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:00.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:00.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:00.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:00.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 152 = '' 2026-02-20T22:50:00.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:00.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:50:00.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2026-02-20T22:50:00.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:00.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:00.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:00.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 152 = '' 2026-02-20T22:50:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:00.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:50:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:01.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:50:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.status 2026-02-20T22:50:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2026-02-20T22:50:01.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:01.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=152 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 152 = 152 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:01.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:01.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:01.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:02.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_34': 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249c8d71545 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249c8d71545 2026-02-20T22:50:02.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:24 2026 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:24 2026 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:24 2026 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 4f963951-2e0e-4d72-bfa8-798685ade5d5 2026-02-20T22:50:02.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_34 snap 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.export 2026-02-20T22:50:02.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.export 2026-02-20T22:50:02.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.export 2026-02-20T22:50:02.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.export 2026-02-20T22:50:02.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:02.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.export 2026-02-20T22:50:02.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:02.642 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.export 2026-02-20T22:50:02.699 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_34.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_34.export 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_35 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_35 snap 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:02.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:04.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2026-02-20T22:50:04.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:04.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:04.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:04.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:05.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:05.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2026-02-20T22:50:05.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:05.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:05.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:05.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2026-02-20T22:50:05.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:05.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2026-02-20T22:50:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-20T22:50:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:05.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2026-02-20T22:50:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:50:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:05.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:50:05.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:50:05.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:50:05.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:50:05.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:05.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2026-02-20T22:50:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2026-02-20T22:50:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:50:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:50:05.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:50:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:05.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2026-02-20T22:50:05.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:05.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:05.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:05.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_35 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:05.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:05.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2026-02-20T22:50:06.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 154 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_35 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:06.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_35.status 2026-02-20T22:50:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_35.status 2026-02-20T22:50:06.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:06.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.status 2026-02-20T22:50:06.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_35.status 2026-02-20T22:50:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_35.status 2026-02-20T22:50:06.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.status 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.status 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2026-02-20T22:50:06.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:06.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=154 2026-02-20T22:50:06.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:06.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:06.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:06.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:06.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:06.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:06.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2026-02-20T22:50:06.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:07.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:07.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=69 2026-02-20T22:50:07.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 154 = 69 2026-02-20T22:50:07.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:07.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:07.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:07.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:07.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2026-02-20T22:50:07.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=69 2026-02-20T22:50:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 154 = 69 2026-02-20T22:50:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:07.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2026-02-20T22:50:08.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:08.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:08.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:08.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 154 = '' 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:08.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:09.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:09.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:09.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:09.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:09.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.status 2026-02-20T22:50:09.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2026-02-20T22:50:09.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:09.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=154 2026-02-20T22:50:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 154 = 154 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:10.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2026-02-20T22:50:10.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_35': 2026-02-20T22:50:10.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:10.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:10.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253ebe44643 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253ebe44643 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:39 2026 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:39 2026 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:39 2026 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:10.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7ab82fe0-366c-45d8-adf9-a17acd1015ea 2026-02-20T22:50:10.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_35 snap 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.export 2026-02-20T22:50:10.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.export 2026-02-20T22:50:10.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.export 2026-02-20T22:50:10.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.export 2026-02-20T22:50:11.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:11.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.export 2026-02-20T22:50:11.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:11.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.export 2026-02-20T22:50:11.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_35.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_35.export 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_36 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_36 snap 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:11.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:11.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:11.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:11.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:12.817 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.532040070301372) > weight (0.25) 2026-02-20T22:50:12.818 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-20T22:50:12.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-20T22:50:12.818 INFO:tasks.rbd_mirror_thrash:waiting for 0.2021937284406483 secs before reviving daemons 2026-02-20T22:50:12.819 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T22:50:12.812+0000 7fc8024f5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 27703) UID: 1000 2026-02-20T22:50:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2026-02-20T22:50:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:12.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:12.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:13.021 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-20T22:50:13.021 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:50:13.025 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-20T22:50:13.025 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-20T22:50:13.026 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-20T22:50:13.072 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-20T22:50:13.072 INFO:tasks.rbd_mirror_thrash:waiting for 72.82123930753261 secs before thrashing 2026-02-20T22:50:13.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:13.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2026-02-20T22:50:13.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:13.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:13.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:13.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:13.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:13.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2026-02-20T22:50:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-20T22:50:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:13.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2026-02-20T22:50:14.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:14.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:50:14.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:50:14.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:14.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2026-02-20T22:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-20T22:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:14.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:50:14.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:50:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:50:14.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:50:14.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:50:14.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:50:14.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2026-02-20T22:50:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2026-02-20T22:50:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:14.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:14.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:14.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:50:14.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:14.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:50:14.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:50:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:50:14.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:50:14.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:50:14.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2026-02-20T22:50:14.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:14.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:14.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_36 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:14.233 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2026-02-20T22:50:14.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 156 2026-02-20T22:50:14.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_36 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:14.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_36.status 2026-02-20T22:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_36.status 2026-02-20T22:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:14.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.status 2026-02-20T22:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_36.status 2026-02-20T22:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_36.status 2026-02-20T22:50:14.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:14.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.status 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.status 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2026-02-20T22:50:14.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:14.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=156 2026-02-20T22:50:14.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:14.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:15.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:15.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2026-02-20T22:50:15.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:15.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=70 2026-02-20T22:50:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 156 = 70 2026-02-20T22:50:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:15.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2026-02-20T22:50:15.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:15.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:15.844 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 156 = '' 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:15.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:16.649 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2026-02-20T22:50:16.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:16.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:16.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 156 = '' 2026-02-20T22:50:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:16.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:16.886 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.876+0000 7f7ed1951640 -1 reset not still connected to 0x55c7096a5450 2026-02-20T22:50:16.887 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.876+0000 7f7ed1951640 -1 reset not still connected to 0x55c70c2c5380 2026-02-20T22:50:16.887 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.876+0000 7f7ed1951640 -1 reset not still connected to 0x55c70c62c820 2026-02-20T22:50:16.887 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.876+0000 7f7ed1951640 -1 reset not still connected to 0x55c70de580d0 2026-02-20T22:50:16.888 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.880+0000 7f7ed1951640 -1 reset not still connected to 0x55c70ee6dba0 2026-02-20T22:50:16.888 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.880+0000 7f7ed1951640 -1 reset not still connected to 0x55c72112a340 2026-02-20T22:50:16.889 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.880+0000 7f7ed1951640 -1 reset not still connected to 0x55c722c8e8f0 2026-02-20T22:50:16.890 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.880+0000 7f7ed1951640 -1 reset not still connected to 0x55c72a527a00 2026-02-20T22:50:16.890 INFO:tasks.ceph.cluster2.osd.2.trial107.stderr:2026-02-20T22:50:16.880+0000 7f7ed1951640 -1 reset not still connected to 0x55c72a527d40 2026-02-20T22:50:18.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.status 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2026-02-20T22:50:18.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:18.315 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:18.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=156 2026-02-20T22:50:18.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 156 = 156 2026-02-20T22:50:18.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:18.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:18.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:19.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2026-02-20T22:50:19.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_36': 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12626a6623d8 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12626a6623d8 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:19.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:41:54 2026 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:41:54 2026 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:41:54 2026 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 685defec-9ba9-4ca2-8daa-bf3c9cad1fd1 2026-02-20T22:50:19.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_36 snap 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.export 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.export 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.export 2026-02-20T22:50:19.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.export 2026-02-20T22:50:19.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.export 2026-02-20T22:50:19.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:19.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.export 2026-02-20T22:50:19.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_36.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_36.export 2026-02-20T22:50:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_37 2026-02-20T22:50:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_37 snap 2026-02-20T22:50:19.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:19.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:20.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:21.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:21.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:21.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:21.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2026-02-20T22:50:22.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:22.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:22.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:22.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:22.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:22.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_37 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:22.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2026-02-20T22:50:22.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:22.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:22.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:22.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2026-02-20T22:50:22.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 158 2026-02-20T22:50:22.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_37 2026-02-20T22:50:22.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_37.status 2026-02-20T22:50:22.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_37.status 2026-02-20T22:50:22.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:22.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.status 2026-02-20T22:50:22.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_37.status 2026-02-20T22:50:22.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_37.status 2026-02-20T22:50:22.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:22.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:22.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.status 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.status 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2026-02-20T22:50:22.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:22.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=158 2026-02-20T22:50:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:23.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2026-02-20T22:50:23.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:23.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:23.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=71 2026-02-20T22:50:23.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 158 = 71 2026-02-20T22:50:23.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:23.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:23.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:23.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:23.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:23.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:23.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:23.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2026-02-20T22:50:23.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:23.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=71 2026-02-20T22:50:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 158 = 71 2026-02-20T22:50:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:23.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:24.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:24.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:24.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:24.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:24.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:24.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2026-02-20T22:50:24.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:24.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:24.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:24.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:24.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:24.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 158 = '' 2026-02-20T22:50:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:24.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.status 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2026-02-20T22:50:26.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:26.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=158 2026-02-20T22:50:26.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 158 = 158 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:26.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:27.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_37': 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213e00ec15c 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213e00ec15c 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:08 2026 2026-02-20T22:50:27.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:08 2026 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:08 2026 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1ca65d24-3943-4fec-a288-c4a0270f5328 2026-02-20T22:50:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_37 snap 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.export 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.export 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.export 2026-02-20T22:50:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.export 2026-02-20T22:50:27.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:27.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.export 2026-02-20T22:50:27.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:27.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.export 2026-02-20T22:50:27.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_37.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_37.export 2026-02-20T22:50:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_38 2026-02-20T22:50:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_38 snap 2026-02-20T22:50:27.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:27.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:27.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:27.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:27.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:28.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:28.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:28.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:29.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2026-02-20T22:50:29.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:29.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:29.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:29.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:29.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:29.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:29.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:29.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:29.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:29.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2026-02-20T22:50:30.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:30.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:30.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:30.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:30.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_38 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:30.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:30.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2026-02-20T22:50:30.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 160 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_38 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:30.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_38.status 2026-02-20T22:50:30.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_38.status 2026-02-20T22:50:30.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:30.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.status 2026-02-20T22:50:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_38.status 2026-02-20T22:50:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_38.status 2026-02-20T22:50:30.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:30.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.status 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.status 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2026-02-20T22:50:30.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:31.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=160 2026-02-20T22:50:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:31.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:31.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:31.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:31.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:31.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:31.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2026-02-20T22:50:31.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:31.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:31.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=72 2026-02-20T22:50:31.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 160 = 72 2026-02-20T22:50:31.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:31.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:31.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2026-02-20T22:50:31.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:31.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=72 2026-02-20T22:50:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 160 = 72 2026-02-20T22:50:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:31.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2026-02-20T22:50:32.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:32.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:32.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:32.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:32.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:32.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 160 = '' 2026-02-20T22:50:32.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:32.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:34.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:34.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:34.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:34.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:34.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.status 2026-02-20T22:50:34.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2026-02-20T22:50:34.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:34.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=160 2026-02-20T22:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 160 = 160 2026-02-20T22:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:34.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:35.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2026-02-20T22:50:35.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_38': 2026-02-20T22:50:35.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249ada84ec5 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249ada84ec5 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:18 2026 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:18 2026 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:18 2026 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 12d9288b-1872-49ea-be06-e6e80ff2ebff 2026-02-20T22:50:35.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_38 snap 2026-02-20T22:50:35.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.export 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.export 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.export 2026-02-20T22:50:35.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.export 2026-02-20T22:50:35.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:35.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.export 2026-02-20T22:50:35.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:35.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.export 2026-02-20T22:50:35.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_38.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_38.export 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_39 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_39 snap 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:35.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:35.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2026-02-20T22:50:36.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:36.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:37.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:37.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:37.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:37.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:37.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:37.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2026-02-20T22:50:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2026-02-20T22:50:38.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:38.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:38.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:38.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2026-02-20T22:50:38.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:38.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:50:38.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:50:38.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:38.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2026-02-20T22:50:38.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:38.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:38.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:38.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:38.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:38.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:38.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_39 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2026-02-20T22:50:38.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:38.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:38.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:38.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2026-02-20T22:50:38.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 162 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_39 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:38.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_39.status 2026-02-20T22:50:38.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_39.status 2026-02-20T22:50:38.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:38.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.status 2026-02-20T22:50:38.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_39.status 2026-02-20T22:50:38.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_39.status 2026-02-20T22:50:38.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.status 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.status 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2026-02-20T22:50:38.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:39.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=162 2026-02-20T22:50:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:39.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2026-02-20T22:50:39.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:39.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=73 2026-02-20T22:50:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 162 = 73 2026-02-20T22:50:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2026-02-20T22:50:39.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:39.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=73 2026-02-20T22:50:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 162 = 73 2026-02-20T22:50:39.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:39.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:40.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:40.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2026-02-20T22:50:40.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:40.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:40.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:40.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 162 = '' 2026-02-20T22:50:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:40.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:42.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.status 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2026-02-20T22:50:42.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:42.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=162 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 162 = 162 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:42.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:43.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2026-02-20T22:50:43.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_39': 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253deaf9bea 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253deaf9bea 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:32 2026 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:32 2026 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:32 2026 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 403c7020-8ef5-46bd-81c8-d009d39aabd9 2026-02-20T22:50:43.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_39 snap 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.export 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.export 2026-02-20T22:50:43.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.export 2026-02-20T22:50:43.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.export 2026-02-20T22:50:43.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:43.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.export 2026-02-20T22:50:43.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:43.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.export 2026-02-20T22:50:43.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_39.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_39.export 2026-02-20T22:50:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:43.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_40 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_40 snap 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:43.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:44.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:44.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2026-02-20T22:50:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:45.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:45.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:45.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:45.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:45.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:45.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:45.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2026-02-20T22:50:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2026-02-20T22:50:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:50:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:50:45.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:45.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:50:45.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:45.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:50:45.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2026-02-20T22:50:45.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:45.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:45.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:45.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:45.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_40 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:45.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:45.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2026-02-20T22:50:46.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 164 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_40 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:46.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_40.status 2026-02-20T22:50:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_40.status 2026-02-20T22:50:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:46.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.status 2026-02-20T22:50:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_40.status 2026-02-20T22:50:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_40.status 2026-02-20T22:50:46.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:46.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.status 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.status 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2026-02-20T22:50:46.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:46.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:46.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=164 2026-02-20T22:50:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:46.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:46.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:46.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:46.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:46.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:46.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:46.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2026-02-20T22:50:46.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:47.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:47.231 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=74 2026-02-20T22:50:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 164 = 74 2026-02-20T22:50:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:47.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:47.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2026-02-20T22:50:47.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:47.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=74 2026-02-20T22:50:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 164 = 74 2026-02-20T22:50:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:47.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2026-02-20T22:50:48.466 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:48.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:48.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:48.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:48.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:48.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 164 = '' 2026-02-20T22:50:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:48.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:50.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.status 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2026-02-20T22:50:50.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:50.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=164 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 164 = 164 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:50.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:50.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:50.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:50.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:50.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:50.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:51.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_40': 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1262f2be4e9e 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1262f2be4e9e 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:38 2026 2026-02-20T22:50:51.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:38 2026 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:38 2026 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 2ce84f99-cd8f-4fec-ba8c-66a66d118bbc 2026-02-20T22:50:51.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_40 snap 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.export 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.export 2026-02-20T22:50:51.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.export 2026-02-20T22:50:51.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.export 2026-02-20T22:50:51.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:51.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.export 2026-02-20T22:50:51.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:51.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.export 2026-02-20T22:50:51.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_40.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_40.export 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_41 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_41 snap 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:51.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:52.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:50:53.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:50:53.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:50:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:50:53.754 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:50:53.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:50:53.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:50:53.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2026-02-20T22:50:53.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:50:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:50:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:50:53.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:50:53.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_41 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:53.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2026-02-20T22:50:54.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 166 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_41 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:50:54.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_41.status 2026-02-20T22:50:54.769 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_41.status 2026-02-20T22:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.status 2026-02-20T22:50:54.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_41.status 2026-02-20T22:50:54.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_41.status 2026-02-20T22:50:54.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.status 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:54.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.status 2026-02-20T22:50:54.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2026-02-20T22:50:54.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:54.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:50:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=166 2026-02-20T22:50:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:54.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:50:55.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:55.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:55.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:55.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:55.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:55.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2026-02-20T22:50:55.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:55.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=75 2026-02-20T22:50:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 166 = 75 2026-02-20T22:50:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:55.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:50:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:55.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:55.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2026-02-20T22:50:55.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:55.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=75 2026-02-20T22:50:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 166 = 75 2026-02-20T22:50:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:55.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:56.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2026-02-20T22:50:56.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:56.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:50:56.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:50:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:50:56.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 166 = '' 2026-02-20T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:50:56.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:50:57.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:57.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:57.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:57.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:57.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.status 2026-02-20T22:50:57.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2026-02-20T22:50:57.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:50:57.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:50:57.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=166 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 166 = 166 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:50:57.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:50:58.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2026-02-20T22:50:58.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_41': 2026-02-20T22:50:58.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213c5eb732c 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213c5eb732c 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:48 2026 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:48 2026 2026-02-20T22:50:58.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:48 2026 2026-02-20T22:50:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:50:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:50:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:50:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 73979901-0e9a-4163-876a-7c36d69ac17d 2026-02-20T22:50:58.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:50:58.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_41 snap 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.export 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.export 2026-02-20T22:50:58.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.export 2026-02-20T22:50:58.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.export 2026-02-20T22:50:59.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:59.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.export 2026-02-20T22:50:59.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:50:59.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.export 2026-02-20T22:50:59.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_41.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_41.export 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_42 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_42 snap 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:50:59.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:00.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2026-02-20T22:51:00.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:00.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:00.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:00.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:00.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:01.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:01.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2026-02-20T22:51:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:01.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:01.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:01.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:01.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2026-02-20T22:51:01.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:01.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:01.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:01.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:01.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_42 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:01.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2026-02-20T22:51:02.778 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 168 2026-02-20T22:51:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_42 2026-02-20T22:51:02.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_42.status 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_42.status 2026-02-20T22:51:02.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:02.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.status 2026-02-20T22:51:02.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_42.status 2026-02-20T22:51:02.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_42.status 2026-02-20T22:51:02.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.status 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:02.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.status 2026-02-20T22:51:02.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2026-02-20T22:51:02.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:02.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:02.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=168 2026-02-20T22:51:02.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:02.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:02.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2026-02-20T22:51:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:03.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:03.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=76 2026-02-20T22:51:03.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 168 = 76 2026-02-20T22:51:03.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:03.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:03.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2026-02-20T22:51:03.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:03.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=76 2026-02-20T22:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 168 = 76 2026-02-20T22:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:03.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2026-02-20T22:51:04.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:04.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:04.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:04.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:04.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:04.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:04.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:04.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 168 = '' 2026-02-20T22:51:04.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:04.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:04.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.30614066028968656) > weight (0.25) 2026-02-20T22:51:04.326 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.27467296460479196) > weight (0.25) 2026-02-20T22:51:04.326 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-20T22:51:04.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-20T22:51:04.327 INFO:tasks.rbd_mirror_thrash:waiting for 6.719808235891217 secs before reviving daemons 2026-02-20T22:51:04.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:51:04.324+0000 7f283595d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 33802) UID: 1000 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.status 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2026-02-20T22:51:05.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:05.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=168 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 168 = 168 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:05.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:05.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:06.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_42': 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249ef6caa31 2026-02-20T22:51:06.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249ef6caa31 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:42:56 2026 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:42:56 2026 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:42:56 2026 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 3b9366c3-0cd7-4131-958f-35709e2da94a 2026-02-20T22:51:06.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_42 snap 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:06.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.export 2026-02-20T22:51:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.export 2026-02-20T22:51:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.export 2026-02-20T22:51:06.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.export 2026-02-20T22:51:07.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:07.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.export 2026-02-20T22:51:07.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:07.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.export 2026-02-20T22:51:07.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_42.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_42.export 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_43 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_43 snap 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:07.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:07.422 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:08.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2026-02-20T22:51:08.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:08.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:09.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2026-02-20T22:51:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:09.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:09.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:09.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:09.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:09.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:09.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:09.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:09.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:09.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2026-02-20T22:51:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-20T22:51:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:09.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_43 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:09.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:09.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:09.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:09.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:09.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:09.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_43 2026-02-20T22:51:09.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_43'] 2026-02-20T22:51:09.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:09.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:09.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:09.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_43 2026-02-20T22:51:09.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:09.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:09.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:09.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:09.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:09.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:09.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:09.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_43 2026-02-20T22:51:10.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:10.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:10.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:10.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_43 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:10.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:10.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2026-02-20T22:51:10.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 170 2026-02-20T22:51:10.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_43 2026-02-20T22:51:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:10.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:10.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_43.status 2026-02-20T22:51:10.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_43.status 2026-02-20T22:51:10.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.status 2026-02-20T22:51:10.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_43.status 2026-02-20T22:51:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_43.status 2026-02-20T22:51:10.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.status 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.status 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2026-02-20T22:51:10.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:10.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:10.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=170 2026-02-20T22:51:10.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:10.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:10.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:11.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2026-02-20T22:51:11.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:11.047 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-20T22:51:11.047 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:51:11.047 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-20T22:51:11.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-20T22:51:11.048 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-20T22:51:11.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-20T22:51:11.050 INFO:tasks.rbd_mirror_thrash:waiting for 64.77607132234996 secs before thrashing 2026-02-20T22:51:11.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=77 2026-02-20T22:51:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 170 = 77 2026-02-20T22:51:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:11.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2026-02-20T22:51:11.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:11.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=77 2026-02-20T22:51:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 170 = 77 2026-02-20T22:51:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:11.510 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:12.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2026-02-20T22:51:12.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:12.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:12.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:12.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:12.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 170 = '' 2026-02-20T22:51:12.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:12.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:13.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.status 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2026-02-20T22:51:13.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:13.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=170 2026-02-20T22:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 170 = 170 2026-02-20T22:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:13.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:13.985 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:14.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2026-02-20T22:51:15.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_43': 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12532914c88e 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12532914c88e 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:02 2026 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:02 2026 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:02 2026 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ba45bb6e-d551-4afb-a2c1-3248377995b6 2026-02-20T22:51:15.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_43 snap 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.export 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.export 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.export 2026-02-20T22:51:15.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.export 2026-02-20T22:51:15.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:15.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.export 2026-02-20T22:51:15.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:15.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.export 2026-02-20T22:51:15.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_43.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_43.export 2026-02-20T22:51:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:15.426 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_44 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_44 snap 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:15.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:16.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:17.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:17.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:17.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:17.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2026-02-20T22:51:17.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:17.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:17.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:17.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:17.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2026-02-20T22:51:17.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-20T22:51:17.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:17.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:17.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:17.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:17.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2026-02-20T22:51:17.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-20T22:51:17.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:17.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_44 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:17.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:17.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:17.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:17.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:17.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:17.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:17.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_44 2026-02-20T22:51:18.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2026-02-20T22:51:18.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:18.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_44 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:18.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:18.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:18.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:18.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:18.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_44 2026-02-20T22:51:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:18.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:18.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:18.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:18.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_44 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:18.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:18.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2026-02-20T22:51:18.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 172 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_44 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_44.status 2026-02-20T22:51:18.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_44.status 2026-02-20T22:51:18.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:18.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.status 2026-02-20T22:51:18.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_44.status 2026-02-20T22:51:18.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_44.status 2026-02-20T22:51:18.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.status 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:18.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.status 2026-02-20T22:51:18.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2026-02-20T22:51:18.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:18.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=172 2026-02-20T22:51:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:18.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:19.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2026-02-20T22:51:19.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:19.089 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:19.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=78 2026-02-20T22:51:19.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 172 = 78 2026-02-20T22:51:19.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:19.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:19.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2026-02-20T22:51:19.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:19.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=78 2026-02-20T22:51:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 172 = 78 2026-02-20T22:51:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:19.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:20.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2026-02-20T22:51:20.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:20.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:20.360 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:20.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 172 = '' 2026-02-20T22:51:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:20.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.status 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2026-02-20T22:51:21.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:21.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=172 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 172 = 172 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:22.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:23.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_44': 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1262a7bd5ebf 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1262a7bd5ebf 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:23.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:12 2026 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:12 2026 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:12 2026 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 9e589007-be33-45e3-a2f7-22c31b17078f 2026-02-20T22:51:23.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_44 snap 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:23.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.export 2026-02-20T22:51:23.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.export 2026-02-20T22:51:23.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.export 2026-02-20T22:51:23.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.export 2026-02-20T22:51:23.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:23.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.export 2026-02-20T22:51:23.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:23.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.export 2026-02-20T22:51:23.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_44.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_44.export 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_45 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_45 snap 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:23.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:24.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2026-02-20T22:51:24.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:24.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:24.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:24.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2026-02-20T22:51:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:25.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:25.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:25.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:25.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:25.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:25.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:25.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:25.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:25.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2026-02-20T22:51:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:25.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:25.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7856556235235578) > weight (0.25) 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7330100266922827) > weight (0.25) 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.9876856177644572) > weight (0.25) 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror_thrash:waiting for 6.915600934595138 secs before reviving daemons 2026-02-20T22:51:25.894 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T22:51:25.891+0000 7fce222d5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 31289) UID: 1000 2026-02-20T22:51:25.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:25.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:25.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:25.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2026-02-20T22:51:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:25.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:25.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:25.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:25.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:25.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:25.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2026-02-20T22:51:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:26.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:26.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:26.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:26.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:26.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:26.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:26.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2026-02-20T22:51:26.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-20T22:51:26.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:26.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 1 2026-02-20T22:51:27.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:27.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:27.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:27.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2026-02-20T22:51:27.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:27.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:27.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:27.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:27.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2026-02-20T22:51:27.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:27.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:27.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:27.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:27.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:27.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:27.184 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2026-02-20T22:51:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2026-02-20T22:51:27.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:27.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:27.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:27.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:27.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:27.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:27.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2026-02-20T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:27.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2026-02-20T22:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:27.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:27.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:27.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:27.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:27.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:27.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:27.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2026-02-20T22:51:27.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-20T22:51:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 2 2026-02-20T22:51:29.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:29.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2026-02-20T22:51:29.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:29.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:29.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:29.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2026-02-20T22:51:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:29.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:29.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:29.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:29.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:29.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2026-02-20T22:51:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:29.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:29.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:29.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:29.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:29.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:29.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:29.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2026-02-20T22:51:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:29.597 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:29.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:29.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:29.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:29.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:29.611 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:29.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2026-02-20T22:51:29.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2026-02-20T22:51:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:29.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 4 2026-02-20T22:51:32.810 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-20T22:51:32.811 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:51:32.811 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-20T22:51:32.812 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:51:32.812 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:51:32.814 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:51:32.814 INFO:tasks.rbd_mirror_thrash:waiting for 95.81489429356978 secs before thrashing 2026-02-20T22:51:33.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:33.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:33.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:33.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2026-02-20T22:51:33.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:33.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:33.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:33.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:33.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:33.673 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:33.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2026-02-20T22:51:33.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:33.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:33.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2026-02-20T22:51:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:33.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:33.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:33.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:33.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:33.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:33.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:33.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2026-02-20T22:51:33.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2026-02-20T22:51:33.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:33.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:33.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:33.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2026-02-20T22:51:33.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:33.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:33.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:33.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:33.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2026-02-20T22:51:33.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:33.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:33.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:33.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_45 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2026-02-20T22:51:33.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:33.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2026-02-20T22:51:34.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 174 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_45 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:34.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_45.status 2026-02-20T22:51:34.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_45.status 2026-02-20T22:51:34.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:34.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.status 2026-02-20T22:51:34.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_45.status 2026-02-20T22:51:34.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_45.status 2026-02-20T22:51:34.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:34.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:34.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.status 2026-02-20T22:51:34.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:34.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.status 2026-02-20T22:51:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2026-02-20T22:51:34.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:34.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:34.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=174 2026-02-20T22:51:34.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:34.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:34.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:35.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2026-02-20T22:51:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:35.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=79 2026-02-20T22:51:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 174 = 79 2026-02-20T22:51:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:35.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:35.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2026-02-20T22:51:35.508 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:35.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:35.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=79 2026-02-20T22:51:35.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 174 = 79 2026-02-20T22:51:35.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:35.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:36.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:36.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2026-02-20T22:51:36.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:36.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:36.373 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:36.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:36.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:36.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 174 = '' 2026-02-20T22:51:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:36.376 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.status 2026-02-20T22:51:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2026-02-20T22:51:37.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:38.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=174 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 174 = 174 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:38.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:38.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:38.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:38.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:38.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:39.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_45': 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213f225fdf9 2026-02-20T22:51:39.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213f225fdf9 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:27 2026 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:27 2026 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:27 2026 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:39.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: ebe686cb-a3fc-48a1-a990-521e037b144f 2026-02-20T22:51:39.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_45 snap 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:39.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.export 2026-02-20T22:51:39.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.export 2026-02-20T22:51:39.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.export 2026-02-20T22:51:39.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.export 2026-02-20T22:51:39.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:39.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.export 2026-02-20T22:51:39.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:39.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.export 2026-02-20T22:51:39.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_45.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_45.export 2026-02-20T22:51:39.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_46 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_46 snap 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:39.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2026-02-20T22:51:40.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:40.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:41.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:41.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:41.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:41.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:41.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:41.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:41.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2026-02-20T22:51:41.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:41.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:41.842 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:41.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:41.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2026-02-20T22:51:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-20T22:51:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:41.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:41.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_46 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:41.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:41.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:41.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:41.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_46 2026-02-20T22:51:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-20T22:51:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_46 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:41.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:41.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:42.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:42.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:42.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_46 2026-02-20T22:51:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_46'] 2026-02-20T22:51:42.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:42.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_46 2026-02-20T22:51:42.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:42.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:42.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:42.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:42.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:42.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_46 2026-02-20T22:51:42.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:42.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:42.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_46 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:42.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2026-02-20T22:51:42.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 176 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_46 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:42.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_46.status 2026-02-20T22:51:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_46.status 2026-02-20T22:51:42.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:42.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.status 2026-02-20T22:51:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_46.status 2026-02-20T22:51:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_46.status 2026-02-20T22:51:42.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.status 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.status 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2026-02-20T22:51:42.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:42.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=176 2026-02-20T22:51:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:42.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2026-02-20T22:51:43.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:43.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=80 2026-02-20T22:51:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 176 = 80 2026-02-20T22:51:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:43.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:43.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:43.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:43.522 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2026-02-20T22:51:43.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:43.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:43.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=80 2026-02-20T22:51:43.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 176 = 80 2026-02-20T22:51:43.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:43.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:44.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:44.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2026-02-20T22:51:44.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:44.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:44.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:44.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:44.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:44.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 176 = '' 2026-02-20T22:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:44.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.status 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2026-02-20T22:51:45.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=176 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 176 = 176 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:46.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:46.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:47.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_46': 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1249cec7fa51 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1249cec7fa51 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:38 2026 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:38 2026 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:38 2026 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 0f4136cd-d44e-415e-a02d-e8635f545728 2026-02-20T22:51:47.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_46 snap 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.export 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.export 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.export 2026-02-20T22:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.export 2026-02-20T22:51:47.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:47.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.export 2026-02-20T22:51:47.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:47.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.export 2026-02-20T22:51:47.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_46.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_46.export 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_47 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_47 snap 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:47.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2026-02-20T22:51:48.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:48.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:49.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2026-02-20T22:51:49.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:49.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:49.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:49.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:49.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:49.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:49.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:49.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:49.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2026-02-20T22:51:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-20T22:51:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:49.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:49.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:49.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:49.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:49.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:49.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2026-02-20T22:51:50.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-20T22:51:50.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:50.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:50.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:2 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:2 cluster instance 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:2 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=2 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:50.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=2 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:2 cluster1 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:2 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:2 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:50.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=2 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 2 = cluster1:2 ']' 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=2 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=2 2026-02-20T22:51:50.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2026-02-20T22:51:50.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:50.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:50.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2026-02-20T22:51:50.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2026-02-20T22:51:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2026-02-20T22:51:50.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:50.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2026-02-20T22:51:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:3 2026-02-20T22:51:50.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:3 cluster instance 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:3 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=3 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=3 2026-02-20T22:51:50.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:3 cluster1 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:3 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:3 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=3 2026-02-20T22:51:50.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 3 = cluster1:3 ']' 2026-02-20T22:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=3 2026-02-20T22:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=3 2026-02-20T22:51:50.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2026-02-20T22:51:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2026-02-20T22:51:50.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2026-02-20T22:51:50.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:50.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:50.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_47 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:50.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:50.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2026-02-20T22:51:50.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 178 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_47 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:50.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_47.status 2026-02-20T22:51:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_47.status 2026-02-20T22:51:50.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:50.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.status 2026-02-20T22:51:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_47.status 2026-02-20T22:51:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_47.status 2026-02-20T22:51:50.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.status 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.status 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2026-02-20T22:51:50.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:50.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=178 2026-02-20T22:51:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:50.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:51.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2026-02-20T22:51:51.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:51.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:51.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=81 2026-02-20T22:51:51.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 178 = 81 2026-02-20T22:51:51.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:51.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2026-02-20T22:51:51.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:51.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:51.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=81 2026-02-20T22:51:51.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 178 = 81 2026-02-20T22:51:51.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:51.572 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:52.374 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2026-02-20T22:51:52.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:52.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:52.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:51:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:51:52.408 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:51:52.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:51:52.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 178 = '' 2026-02-20T22:51:52.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:52.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:51:54.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:54.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.status 2026-02-20T22:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2026-02-20T22:51:54.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:54.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:54.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=178 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 178 = 178 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:54.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:51:54.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:55.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_47': 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1253f23758e0 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1253f23758e0 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:51:55.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:47 2026 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:47 2026 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:47 2026 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: a7ff2d10-0af9-4bfa-8665-8b032dd2395c 2026-02-20T22:51:55.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_47 snap 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:55.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:51:55.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.export 2026-02-20T22:51:55.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.export 2026-02-20T22:51:55.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.export 2026-02-20T22:51:55.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.export 2026-02-20T22:51:55.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:55.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.export 2026-02-20T22:51:55.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:51:55.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.export 2026-02-20T22:51:55.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_47.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_47.export 2026-02-20T22:51:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:51:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_48 2026-02-20T22:51:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_48 snap 2026-02-20T22:51:55.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:51:55.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:51:56.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2026-02-20T22:51:56.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:56.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:56.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:56.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:56.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:51:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2026-02-20T22:51:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:57.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:57.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:51:57.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:51:57.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2026-02-20T22:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:57.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:51:57.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:51:57.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:51:57.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:51:57.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:51:57.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:51:57.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:51:57.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2026-02-20T22:51:57.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2026-02-20T22:51:57.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:51:57.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:51:57.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:51:57.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:51:57.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:51:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:57.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:57.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:51:57.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2026-02-20T22:51:58.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:51:58.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:51:58.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:51:58.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:51:58.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:51:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:51:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:51:58.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_48 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:58.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:58.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:58.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2026-02-20T22:51:58.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 180 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_48 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_48.status 2026-02-20T22:51:58.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_48.status 2026-02-20T22:51:58.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:58.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.status 2026-02-20T22:51:58.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_48.status 2026-02-20T22:51:58.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_48.status 2026-02-20T22:51:58.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.status 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.status 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2026-02-20T22:51:58.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:58.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:51:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=180 2026-02-20T22:51:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:51:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:58.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2026-02-20T22:51:59.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=82 2026-02-20T22:51:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 180 = 82 2026-02-20T22:51:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2026-02-20T22:51:59.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:51:59.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:51:59.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=82 2026-02-20T22:51:59.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 180 = 82 2026-02-20T22:51:59.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:51:59.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2026-02-20T22:52:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:00.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:00.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:52:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:52:00.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:52:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:52:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 180 = '' 2026-02-20T22:52:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:00.424 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.status 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2026-02-20T22:52:02.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:02.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:02.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=180 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 180 = 180 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:02.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:52:02.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:02.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:52:02.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:52:02.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:52:03.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2026-02-20T22:52:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_48': 2026-02-20T22:52:03.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12624554fe05 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12624554fe05 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:43:58 2026 2026-02-20T22:52:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:43:58 2026 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:43:58 2026 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 8397c19c-505f-4fea-8ee6-6d3f6b4a2bbd 2026-02-20T22:52:03.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_48 snap 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.export 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.export 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.export 2026-02-20T22:52:03.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.export 2026-02-20T22:52:03.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:03.244 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.export 2026-02-20T22:52:03.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.export 2026-02-20T22:52:03.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_48.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_48.export 2026-02-20T22:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_49 2026-02-20T22:52:03.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_49 snap 2026-02-20T22:52:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:03.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:52:04.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:52:04.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:52:04.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:52:05.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:52:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:52:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:52:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:52:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:52:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:52:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:05.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2026-02-20T22:52:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2026-02-20T22:52:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd mirror status : get status for rbd mirror 2026-02-20T22:52:05.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:admin_socket: invalid command 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:1 cluster instance 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 1 = cluster1:1 ']' 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:1 cluster1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:52:05.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 1 = cluster1:1 ']' 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=1 2026-02-20T22:52:05.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2026-02-20T22:52:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:52:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:52:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2026-02-20T22:52:05.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2026-02-20T22:52:06.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:52:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:52:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:52:06.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:52:06.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:52:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:52:06.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_49 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2026-02-20T22:52:06.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:06.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2026-02-20T22:52:06.892 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 182 2026-02-20T22:52:06.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_49 2026-02-20T22:52:06.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:52:06.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:52:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:52:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_49.status 2026-02-20T22:52:06.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_49.status 2026-02-20T22:52:06.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:52:06.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.status 2026-02-20T22:52:06.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_49.status 2026-02-20T22:52:06.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_49.status 2026-02-20T22:52:06.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:52:06.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:06.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.status 2026-02-20T22:52:06.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:06.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:06.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:06.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.status 2026-02-20T22:52:06.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2026-02-20T22:52:06.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:06.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=182 2026-02-20T22:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:06.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2026-02-20T22:52:07.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:07.377 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=83 2026-02-20T22:52:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 182 = 83 2026-02-20T22:52:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:07.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:52:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:07.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2026-02-20T22:52:07.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:07.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=83 2026-02-20T22:52:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 182 = 83 2026-02-20T22:52:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:07.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:52:08.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:08.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:08.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:08.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:08.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:08.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2026-02-20T22:52:08.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:08.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:52:08.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:52:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:52:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 182 = '' 2026-02-20T22:52:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:08.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.status 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2026-02-20T22:52:10.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:10.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=182 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 182 = 182 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:10.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:52:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:52:10.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:52:11.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_49': 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1213ce9cc9d2 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1213ce9cc9d2 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:44:06 2026 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:44:06 2026 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:44:06 2026 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: be759c64-ef88-4e79-b56b-60211e4d3d1f 2026-02-20T22:52:11.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:52:11.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_49 snap 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.export 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.export 2026-02-20T22:52:11.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.export 2026-02-20T22:52:11.325 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.export 2026-02-20T22:52:11.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:11.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.export 2026-02-20T22:52:11.660 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:11.670 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.export 2026-02-20T22:52:11.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_49.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_49.export 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_50 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_snap cluster2 mirror image_50 snap 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:11.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2026-02-20T22:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2026-02-20T22:52:12.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=started 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:52:12.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:52:13.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2026-02-20T22:52:13.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=started 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_result 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=stopped 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -i state 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_instance=cluster1 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:52:13.901 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance=cluster1 2026-02-20T22:52:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local loop_instance 2026-02-20T22:52:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2026-02-20T22:52:13.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sleep 0 2026-02-20T22:52:13.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' cluster1 '!=' cluster1 ']' 2026-02-20T22:52:13.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ seq 0 3 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1:0 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ shift 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ set_cluster_instance cluster1:0 cluster instance 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local val=cluster1:0 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster_var_name=cluster 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local instance_var_name=instance 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:52:13.905 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ '[' 0 = cluster1:0 ']' 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval cluster=cluster1 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ eval instance=0 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ daemon_asok_file cluster1:0 cluster1 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local local_cluster=cluster1:0 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster=cluster1 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local val=cluster1:0 2026-02-20T22:52:13.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local cluster_var_name=local_cluster 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ local instance_var_name=instance 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ cluster=cluster1 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ instance=0 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ '[' 0 = cluster1:0 ']' 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval local_cluster=cluster1 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ local_cluster=cluster1 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ eval instance=0 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ instance=0 2026-02-20T22:52:13.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2026-02-20T22:52:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2026-02-20T22:52:13.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2026-02-20T22:52:13.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ return 0 2026-02-20T22:52:13.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ status_result=' "state": "Replaying"' 2026-02-20T22:52:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo ' "state": "Replaying"' 2026-02-20T22:52:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ grep -i Replaying 2026-02-20T22:52:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: "state": "Replaying" 2026-02-20T22:52:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=started 2026-02-20T22:52:13.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test started = started 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror mirror image_50 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:13.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = journal ']' 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' snapshot = snapshot ']' 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2026-02-20T22:52:14.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Snapshot ID: 184 2026-02-20T22:52:14.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror mirror image_50 2026-02-20T22:52:14.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_cluster=cluster1 2026-02-20T22:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_pool=mirror 2026-02-20T22:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local remote_pool=mirror 2026-02-20T22:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:14.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster2-mirror-image_50.status 2026-02-20T22:52:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster2-mirror-image_50.status 2026-02-20T22:52:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:52:14.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local status_log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.status 2026-02-20T22:52:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ mkfname cluster1-mirror-image_50.status 2026-02-20T22:52:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ echo cluster1-mirror-image_50.status 2026-02-20T22:52:14.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -e 's|[/ ]|_|g' 2026-02-20T22:52:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local local_status_log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.status 2026-02-20T22:52:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:14.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.status 2026-02-20T22:52:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2026-02-20T22:52:14.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:14.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/id 2026-02-20T22:52:14.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snapshot_id=184 2026-02-20T22:52:14.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:14.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:14.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.2 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2026-02-20T22:52:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:15.392 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:15.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=84 2026-02-20T22:52:15.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 184 = 84 2026-02-20T22:52:15.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:15.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.4 2026-02-20T22:52:15.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2026-02-20T22:52:15.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:15.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4732558420295855) > weight (0.25) 2026-02-20T22:52:15.827 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9056698405330268) > weight (0.25) 2026-02-20T22:52:15.828 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-20T22:52:15.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-20T22:52:15.828 INFO:tasks.rbd_mirror_thrash:waiting for 9.544925122260901 secs before reviving daemons 2026-02-20T22:52:15.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:15.828 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:15.823+0000 7f98fb75c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 44520) UID: 1000 2026-02-20T22:52:15.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:15.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:52:15.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:52:15.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:52:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:52:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 184 = '' 2026-02-20T22:52:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:15.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.8 2026-02-20T22:52:16.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:16.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2026-02-20T22:52:16.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:16.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:52:16.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:16.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:-:1.1: Document is empty 2026-02-20T22:52:16.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: 2026-02-20T22:52:16.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:^ 2026-02-20T22:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id= 2026-02-20T22:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 184 = '' 2026-02-20T22:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2026-02-20T22:52:16.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1.6 2026-02-20T22:52:18.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local log=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.status 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2026-02-20T22:52:18.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2026-02-20T22:52:18.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2026-02-20T22:52:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local primary_snapshot_id=184 2026-02-20T22:52:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 184 = 184 2026-02-20T22:52:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2026-02-20T22:52:18.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2026-02-20T22:52:18.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 1 2026-02-20T22:52:19.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_50': 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 128 MiB in 32 objects 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 2 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 12497cdcb90e 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.12497cdcb90e 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:52:19.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock, non-primary 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:44:20 2026 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:44:20 2026 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:44:20 2026 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: protected: False 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7389bdfb-9491-4580-9c72-b6dfcd587c3b 2026-02-20T22:52:19.334 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: false 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ compare_image_snaps mirror image_50 snap 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap_name=snap 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local ret=0 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local rmt_export=/tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.export 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local loc_export=/tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.export 2026-02-20T22:52:19.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.export 2026-02-20T22:52:19.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.export 2026-02-20T22:52:19.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:19.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.export 2026-02-20T22:52:19.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2026-02-20T22:52:19.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cmp /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.export 2026-02-20T22:52:19.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -f /tmp/tmp.l7jgOKLHDi/cluster2-mirror-image_50.export /tmp/tmp.l7jgOKLHDi/cluster1-mirror-image_50.export 2026-02-20T22:52:19.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:19.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: delete many images' 2026-02-20T22:52:19.804 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:52:19.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:52:19.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:52:19 TEST: delete many images 2026-02-20T22:52:19.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:52:19 TEST: delete many images 2026-02-20T22:52:19.806 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 50 2026-02-20T22:52:19.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_1 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:19.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2026-02-20T22:52:19.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_1 2026-02-20T22:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:52:19.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:19.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:19.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_1 2026-02-20T22:52:19.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:52:19.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2026-02-20T22:52:19.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2026-02-20T22:52:20.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:52:20.011+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640b380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:20.017 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:52:20.011+0000 7f9327dd7640 -1 rbd::mirror::ImageReplayer: 0x556e0640d180 [3/3ec1d56b-1fff-4c3b-a39e-7c1c401a9ca7] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:20.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_2 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:52:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:20.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2026-02-20T22:52:21.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_2 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:21.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_2 2026-02-20T22:52:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:52:21.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2026-02-20T22:52:21.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2026-02-20T22:52:22.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:22.011+0000 7f8ee9562640 -1 rbd::mirror::ImageMap: 0x558910a73560 handle_peer_ack_remove: failed to remove global_image_id=241a16d7-c45c-4a2f-9678-2850dc15e21b 2026-02-20T22:52:22.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:22.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:22.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:22.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_3 2026-02-20T22:52:22.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2026-02-20T22:52:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:52:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:22.983 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2026-02-20T22:52:23.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:23.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_3 2026-02-20T22:52:23.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:23.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:23.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:52:23.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:23.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:23.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_3 2026-02-20T22:52:23.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:23.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:23.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:52:23.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2026-02-20T22:52:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2026-02-20T22:52:24.015 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:24.011+0000 7f8ee9562640 -1 rbd::mirror::ImageMap: 0x558910a73560 handle_peer_ack_remove: failed to remove global_image_id=2e6164f9-d122-4af4-b228-4bf39a22c6fb 2026-02-20T22:52:24.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:24.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_4 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:52:24.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:25.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2026-02-20T22:52:25.373 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-20T22:52:25.373 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:52:25.374 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-20T22:52:25.374 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-20T22:52:25.374 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-20T22:52:25.376 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-20T22:52:25.376 INFO:tasks.rbd_mirror_thrash:waiting for 95.99413722378594 secs before thrashing 2026-02-20T22:52:25.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_4 2026-02-20T22:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:25.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_4 2026-02-20T22:52:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:52:25.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2026-02-20T22:52:25.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2026-02-20T22:52:26.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:26.039+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589105641a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:26.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:26.039+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558910e65900 [3/ddee4f8b-4aff-4c02-80d5-06e133a89824] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:26.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_5 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:27.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:27.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:52:27.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:27.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2026-02-20T22:52:27.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:27.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_5 2026-02-20T22:52:27.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:27.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:27.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:52:27.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:27.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:27.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_5 2026-02-20T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:52:27.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2026-02-20T22:52:27.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2026-02-20T22:52:28.034 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:28.026+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94ec8b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:28.035 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:28.026+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95768000 [3/6be28159-a310-434e-99b8-33aa6cef5760] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:28.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:28.050+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94ec4b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:28.056 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:28.050+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95768000 [3/6be28159-a310-434e-99b8-33aa6cef5760] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:29.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:29.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:29.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:29.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_6 2026-02-20T22:52:29.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2026-02-20T22:52:29.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:29.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:29.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:52:29.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:29.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2026-02-20T22:52:29.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_6 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:29.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:29.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_6 2026-02-20T22:52:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:52:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2026-02-20T22:52:29.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2026-02-20T22:52:30.026 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:52:30.018+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640b860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:30.027 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:52:30.018+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640c000 [3/1b45a9ef-3d84-4767-b2e5-fe95a04ac4ac] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:31.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_7 2026-02-20T22:52:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2026-02-20T22:52:31.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:52:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:31.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2026-02-20T22:52:31.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:31.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_7 2026-02-20T22:52:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:52:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:31.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:31.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_7 2026-02-20T22:52:31.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:31.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:31.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:52:31.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2026-02-20T22:52:31.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2026-02-20T22:52:32.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:32.038+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95eda680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:32.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:32.038+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95a32c80 [3/9795c60e-64e0-4de7-a6fb-f8429e83e276] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:33.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_8 2026-02-20T22:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2026-02-20T22:52:33.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:33.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:33.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:52:33.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:33.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2026-02-20T22:52:33.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_8 2026-02-20T22:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:52:33.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:33.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_8 2026-02-20T22:52:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:52:33.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2026-02-20T22:52:33.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2026-02-20T22:52:34.062 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:34.054+0000 7f8ee9562640 -1 rbd::mirror::ImageMap: 0x558910a73560 handle_peer_ack_remove: failed to remove global_image_id=25965088-a176-4775-96aa-3e7b420cb588 2026-02-20T22:52:34.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:35.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_9 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:35.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2026-02-20T22:52:35.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_9 2026-02-20T22:52:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:52:35.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:35.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_9 2026-02-20T22:52:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:52:35.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2026-02-20T22:52:35.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2026-02-20T22:52:36.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:36.046+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95edb380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:36.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:36.046+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95768280 [3/eae8172c-87a1-437f-8f5e-44b17669fdc4] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:37.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_10 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:37.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:52:37.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:37.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2026-02-20T22:52:37.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_10 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:37.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_10 2026-02-20T22:52:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:37.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:52:37.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2026-02-20T22:52:37.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2026-02-20T22:52:38.068 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:38.062+0000 7f8ee9562640 -1 rbd::mirror::ImageMap: 0x558910a73560 handle_peer_ack_remove: failed to remove global_image_id=21a4a0c9-3c5e-4e1d-a979-7e282caacbc3 2026-02-20T22:52:39.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_11 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:39.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2026-02-20T22:52:39.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_11 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:39.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_11 2026-02-20T22:52:39.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:39.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:39.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:52:39.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2026-02-20T22:52:39.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2026-02-20T22:52:40.233 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:40.226+0000 7fba48522640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95edb040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:40.234 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:40.226+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95768a00 [3/c1afeba9-ed53-4844-9dd9-34f5c5ee4c22] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:41.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:41.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:41.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:41.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_12 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:41.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2026-02-20T22:52:41.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:41.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_12 2026-02-20T22:52:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:52:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:41.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_12 2026-02-20T22:52:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:52:41.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2026-02-20T22:52:41.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2026-02-20T22:52:42.067 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:42.062+0000 7f8ee8d61640 -1 rbd::mirror::ImageMap: 0x558910a73560 handle_peer_ack_remove: failed to remove global_image_id=97e62ce9-2919-43d0-b90f-ef29433498b6 2026-02-20T22:52:43.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_13 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:43.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2026-02-20T22:52:43.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_13 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:43.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_13 2026-02-20T22:52:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:52:43.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2026-02-20T22:52:44.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2026-02-20T22:52:44.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:44.070+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a6820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:44.078 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:44.070+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b2000 [3/63774277-8720-40a9-a26f-36b0c3afe1c0] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:45.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_14 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:45.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2026-02-20T22:52:45.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_14 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:45.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_14 2026-02-20T22:52:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:45.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:52:45.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2026-02-20T22:52:46.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2026-02-20T22:52:46.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:46.074+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95eda820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:46.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:46.074+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95a33180 [3/1ca8b7ab-8952-4355-92a0-9028c13306a9] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:47.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:47.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_15 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:47.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2026-02-20T22:52:47.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_15 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:47.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_15 2026-02-20T22:52:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:52:47.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2026-02-20T22:52:48.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2026-02-20T22:52:48.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:48.082+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a6340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:48.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:48.082+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b2780 [3/7d4c2418-4884-49bd-8deb-da886a79291a] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:49.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:49.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_16 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:49.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2026-02-20T22:52:49.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_16 2026-02-20T22:52:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:52:49.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_16 2026-02-20T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:49.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:52:49.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2026-02-20T22:52:49.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2026-02-20T22:52:50.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:50.038+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c960ce1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:50.046 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:50.038+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95769400 [3/dd77e808-55a0-4c1a-a3d7-263a1ff1cd0d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:51.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_17 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:51.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2026-02-20T22:52:51.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_17 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:51.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_17 2026-02-20T22:52:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:52:51.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2026-02-20T22:52:52.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2026-02-20T22:52:52.089 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:52.082+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c960ce340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:52.090 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:52:52.082+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95769180 [3/58103425-2768-48ad-acca-e3c306a52862] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:53.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:53.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_18 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:53.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2026-02-20T22:52:53.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:53.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_18 2026-02-20T22:52:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:52:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:53.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_18 2026-02-20T22:52:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:52:53.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2026-02-20T22:52:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2026-02-20T22:52:54.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:54.098+0000 7fc4536ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3868ea1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:54.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:54.098+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b3180 [3/2e7d6018-b8f8-4aa9-a620-9abcba9ad630] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:55.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_19 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:55.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2026-02-20T22:52:55.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_19 2026-02-20T22:52:55.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:52:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:56.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_19 2026-02-20T22:52:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:52:56.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2026-02-20T22:52:56.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2026-02-20T22:52:56.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:56.094+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a6ea0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:56.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:52:56.094+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d386d1db80 [3/3c21132a-e262-46d0-a004-e6e7cbc448c9] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:57.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_20 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:57.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2026-02-20T22:52:57.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_20 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:58.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_20 2026-02-20T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:58.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:52:58.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2026-02-20T22:52:58.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2026-02-20T22:52:58.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:58.102+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558911333d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:52:58.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:52:58.102+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558911056000 [3/84a30fa8-259c-4c04-88f1-8a4e44ec7139] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:52:59.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_21 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:52:59.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2026-02-20T22:52:59.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_21 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:52:59.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:52:59.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_21 2026-02-20T22:52:59.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:52:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:52:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:52:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2026-02-20T22:52:59.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2026-02-20T22:53:00.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:00.042+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589116429c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:00.050 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:00.042+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e65900 [3/69840726-9864-4961-8865-3713146736b0] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:01.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_22 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:01.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2026-02-20T22:53:02.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_22 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:02.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_22 2026-02-20T22:53:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:53:02.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2026-02-20T22:53:02.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2026-02-20T22:53:02.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:02.106+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a7040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:02.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:02.106+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b3400 [3/49060f6a-cc8c-4444-ac78-938dac73f064] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:03.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_23 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:03.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2026-02-20T22:53:04.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_23 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:04.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_23 2026-02-20T22:53:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:53:04.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2026-02-20T22:53:04.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2026-02-20T22:53:04.123 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:04.118+0000 7fc4536ec640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a71e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:04.124 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:04.118+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d386d1d900 [3/357d90ab-5f43-4a44-b85f-74b7bbc1d809] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:05.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_24 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:05.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2026-02-20T22:53:06.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_24 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:06.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_24 2026-02-20T22:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:53:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2026-02-20T22:53:06.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2026-02-20T22:53:06.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:06.106+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558911333a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:06.112 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:06.106+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558910e64c80 [3/eaa9a84f-e88b-4266-b522-69fac662b35d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:07.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_25 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:53:07.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:07.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2026-02-20T22:53:08.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:08.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_25 2026-02-20T22:53:08.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:08.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:08.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:53:08.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:08.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_25 2026-02-20T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:53:08.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2026-02-20T22:53:08.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2026-02-20T22:53:08.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:08.114+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94eca000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:08.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:08.114+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95768780 [3/07156c03-24cc-4962-b7eb-f044afb9ccd1] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:08.630 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3451983132615928) > weight (0.25) 2026-02-20T22:53:08.630 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.39911103019618643) > weight (0.25) 2026-02-20T22:53:08.631 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.7749499178491851) > weight (0.25) 2026-02-20T22:53:08.631 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.29390379864685545) > weight (0.25) 2026-02-20T22:53:08.631 INFO:tasks.rbd_mirror_thrash:waiting for 99.12630610514637 secs before thrashing 2026-02-20T22:53:09.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_26 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:09.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:53:09.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:09.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2026-02-20T22:53:09.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_26 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:09.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_26 2026-02-20T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:53:09.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2026-02-20T22:53:09.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2026-02-20T22:53:10.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:10.038+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a7380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:10.045 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:10.038+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b2a00 [3/96d0ea21-daaa-43c4-b57c-6d4a1854bd5a] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:11.086 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:11.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:11.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:11.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_27 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:11.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2026-02-20T22:53:12.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_27 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:12.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_27 2026-02-20T22:53:12.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:53:12.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2026-02-20T22:53:12.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2026-02-20T22:53:12.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:12.138+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910a7a340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:12.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:12.138+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e64000 [3/9dacbf76-f085-4405-b87d-fe7a44b68efb] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:13.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_28 2026-02-20T22:53:13.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2026-02-20T22:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:13.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2026-02-20T22:53:14.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:14.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_28 2026-02-20T22:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:14.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_28 2026-02-20T22:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:53:14.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2026-02-20T22:53:14.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2026-02-20T22:53:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:14.142+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910565380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:14.147 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:14.142+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e64500 [3/f3e3b67e-63fc-4f0c-a1ad-75bec4cf631a] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:15.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_29 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:15.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2026-02-20T22:53:16.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:16.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_29 2026-02-20T22:53:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:53:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:16.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_29 2026-02-20T22:53:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:53:16.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2026-02-20T22:53:16.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2026-02-20T22:53:16.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:16.130+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910a764e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:16.137 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:16.130+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e64780 [3/b44f2e80-554c-46e7-9cff-22a5d35c7769] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:17.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_30 2026-02-20T22:53:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2026-02-20T22:53:17.108 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:53:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:17.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2026-02-20T22:53:18.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:18.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_30 2026-02-20T22:53:18.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:18.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:18.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:53:18.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:18.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:18.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_30 2026-02-20T22:53:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:53:18.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2026-02-20T22:53:18.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2026-02-20T22:53:18.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:18.138+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a7520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:18.144 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:18.138+0000 7fc459ef9640 -1 rbd::mirror::ImageReplayer: 0x55d3871b2500 [3/76b57b0a-af4f-4c2a-9a97-d9e658cc43f5] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:19.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_31 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:53:19.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:19.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2026-02-20T22:53:19.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_31 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:19.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_31 2026-02-20T22:53:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:19.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:19.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:53:19.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2026-02-20T22:53:19.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2026-02-20T22:53:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:20.038+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e06c9e1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:20.044 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:20.038+0000 7f9327dd7640 -1 rbd::mirror::ImageReplayer: 0x556e05e4d900 [3/0d6df238-c085-4a6e-ad2d-7aa25e3b6e40] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:20.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_32 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2026-02-20T22:53:21.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:21.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_32 2026-02-20T22:53:21.939 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:53:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:21.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:21.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_32 2026-02-20T22:53:21.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:21.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:21.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:53:21.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2026-02-20T22:53:21.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2026-02-20T22:53:22.051 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:22.046+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94ecb380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:22.052 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:22.046+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95769b80 [3/3e048625-939b-485e-ae8c-d4b0fec7ae1d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:23.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:23.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_33 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:23.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2026-02-20T22:53:23.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:23.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_33 2026-02-20T22:53:23.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:23.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:53:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:23.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_33 2026-02-20T22:53:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:53:23.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2026-02-20T22:53:23.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2026-02-20T22:53:24.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:24.034+0000 7f9320dc9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e06c9e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:24.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:24.034+0000 7f9327dd7640 -1 rbd::mirror::ImageReplayer: 0x556e05e4db80 [3/127d9d72-c25f-4d41-a45b-12cfcc97b19d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:25.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_34 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:25.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2026-02-20T22:53:25.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:25.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_34 2026-02-20T22:53:25.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:53:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:25.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_34 2026-02-20T22:53:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:53:25.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2026-02-20T22:53:25.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2026-02-20T22:53:26.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:26.050+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910ce3520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:26.055 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:26.050+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558911056280 [3/4f963951-2e0e-4d72-bfa8-798685ade5d5] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:27.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:27.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:27.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:27.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_35 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:27.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2026-02-20T22:53:27.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:27.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_35 2026-02-20T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:27.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_35 2026-02-20T22:53:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:53:27.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2026-02-20T22:53:27.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2026-02-20T22:53:28.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:28.054+0000 7f9320dc9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640b1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:28.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:28.054+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640d900 [3/7ab82fe0-366c-45d8-adf9-a17acd1015ea] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:29.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:29.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_36 2026-02-20T22:53:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2026-02-20T22:53:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:29.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:53:29.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:29.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2026-02-20T22:53:29.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_36 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:29.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:29.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_36 2026-02-20T22:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:53:29.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2026-02-20T22:53:29.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2026-02-20T22:53:30.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:30.034+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640ba00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:30.041 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:30.034+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640d680 [3/685defec-9ba9-4ca2-8daa-bf3c9cad1fd1] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:31.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_37 2026-02-20T22:53:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2026-02-20T22:53:31.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:53:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:31.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2026-02-20T22:53:31.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:31.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_37 2026-02-20T22:53:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:53:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:31.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_37 2026-02-20T22:53:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:31.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:53:31.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2026-02-20T22:53:32.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2026-02-20T22:53:32.079 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:32.074+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640bd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:32.080 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:32.074+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640c500 [3/1ca65d24-3943-4fec-a288-c4a0270f5328] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:33.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_38 2026-02-20T22:53:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2026-02-20T22:53:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:53:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2026-02-20T22:53:33.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_38 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:33.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_38 2026-02-20T22:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:53:33.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2026-02-20T22:53:34.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2026-02-20T22:53:34.084 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:34.078+0000 7f9320dc9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:34.085 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:34.078+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640cc80 [3/12d9288b-1872-49ea-be06-e6e80ff2ebff] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:35.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:35.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_39 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:35.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2026-02-20T22:53:35.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:35.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_39 2026-02-20T22:53:35.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:35.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:35.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:53:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:35.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_39 2026-02-20T22:53:35.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:35.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:35.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:53:35.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2026-02-20T22:53:36.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2026-02-20T22:53:36.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:36.082+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640ab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:36.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:36.082+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640d400 [3/403c7020-8ef5-46bd-81c8-d009d39aabd9] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:37.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:37.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_40 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:37.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2026-02-20T22:53:37.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_40 2026-02-20T22:53:37.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:53:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_40 2026-02-20T22:53:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:53:37.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2026-02-20T22:53:38.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2026-02-20T22:53:38.088 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:38.082+0000 7f9320dc9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0640a4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:39.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:39.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_41 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:39.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2026-02-20T22:53:39.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:39.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_41 2026-02-20T22:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:39.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:39.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_41 2026-02-20T22:53:39.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:39.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:39.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:53:39.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2026-02-20T22:53:39.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2026-02-20T22:53:40.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:40.050+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589113336c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:40.057 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:40.050+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e65400 [3/73979901-0e9a-4163-876a-7c36d69ac17d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:40.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:40.114+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558911333860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:40.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:40.114+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558910e65400 [3/73979901-0e9a-4163-876a-7c36d69ac17d] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:41.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:41.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:41.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:41.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_42 2026-02-20T22:53:41.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2026-02-20T22:53:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:53:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:41.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2026-02-20T22:53:41.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_42 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:41.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:41.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_42 2026-02-20T22:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:53:41.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2026-02-20T22:53:42.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2026-02-20T22:53:42.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:42.086+0000 7f9320dc9640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e060784e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:42.092 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:42.086+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640ca00 [3/3b9366c3-0cd7-4131-958f-35709e2da94a] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:43.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:43.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_43 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:43.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2026-02-20T22:53:43.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_43 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:43.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_43 2026-02-20T22:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:53:43.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2026-02-20T22:53:44.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2026-02-20T22:53:44.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:44.086+0000 7f93215ca640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x556e0691fa00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:44.094 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:53:44.086+0000 7f93275d6640 -1 rbd::mirror::ImageReplayer: 0x556e0640db80 [3/ba45bb6e-d551-4afb-a2c1-3248377995b6] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:45.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:45.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_44 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:45.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2026-02-20T22:53:45.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_44 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:45.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:45.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_44 2026-02-20T22:53:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:53:45.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2026-02-20T22:53:46.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2026-02-20T22:53:46.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:46.102+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910ce3380 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:46.106 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:46.102+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558911056780 [3/9e589007-be33-45e3-a2f7-22c31b17078f] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:47.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_45 2026-02-20T22:53:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2026-02-20T22:53:47.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:47.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:47.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:53:47.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:47.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2026-02-20T22:53:47.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:48.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_45 2026-02-20T22:53:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:53:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:48.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_45 2026-02-20T22:53:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:53:48.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2026-02-20T22:53:48.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2026-02-20T22:53:48.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:48.102+0000 7fba48522640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94ec9d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:48.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:48.102+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95a32a00 [3/ebe686cb-a3fc-48a1-a990-521e037b144f] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:49.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_46 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:49.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2026-02-20T22:53:49.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_46 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:49.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_46 2026-02-20T22:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:53:49.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2026-02-20T22:53:49.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2026-02-20T22:53:50.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:50.050+0000 7fba48522640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c94ec89c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:50.058 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:50.050+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95769680 [3/0f4136cd-d44e-415e-a02d-e8635f545728] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:51.069 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:51.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:51.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:51.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_47 2026-02-20T22:53:51.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2026-02-20T22:53:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:53:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:51.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2026-02-20T22:53:52.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:52.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_47 2026-02-20T22:53:52.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:52.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_47 2026-02-20T22:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:53:52.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2026-02-20T22:53:52.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2026-02-20T22:53:52.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:52.110+0000 7fba48d23640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95c97d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:52.115 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:52.110+0000 7fba4f530640 -1 rbd::mirror::ImageReplayer: 0x557c95768f00 [3/a7ff2d10-0af9-4bfa-8665-8b032dd2395c] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:53.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:53.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_48 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:53.084 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2026-02-20T22:53:54.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_48 2026-02-20T22:53:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:54.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:53:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:54.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_48 2026-02-20T22:53:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:53:54.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2026-02-20T22:53:54.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2026-02-20T22:53:54.121 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:54.114+0000 7fba48522640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x557c95007860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:54.122 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:53:54.114+0000 7fba4ed2f640 -1 rbd::mirror::ImageReplayer: 0x557c95768c80 [3/8397c19c-505f-4fea-8ee6-6d3f6b4a2bbd] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:55.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_49 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:55.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2026-02-20T22:53:56.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_49 2026-02-20T22:53:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:56.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:53:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:56.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_49 2026-02-20T22:53:56.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:53:56.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2026-02-20T22:53:56.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2026-02-20T22:53:56.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:56.126+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910ce36c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:56.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:53:56.126+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e65180 [3/be759c64-ef88-4e79-b56b-60211e4d3d1f] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:57.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:57.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:57.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:57.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_50 2026-02-20T22:53:57.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2026-02-20T22:53:57.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:57.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:57.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:53:57.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local snap=snap 2026-02-20T22:53:57.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2026-02-20T22:53:58.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing snap: 100% complete...done. 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_50 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:53:58.020 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:53:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_50 2026-02-20T22:53:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:53:58.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:53:58.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2026-02-20T22:53:58.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2026-02-20T22:53:58.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:58.118+0000 7fc452eeb640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55d3871a6d00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:53:58.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:53:58.118+0000 7fc4596f8640 -1 rbd::mirror::ImageReplayer: 0x55d3871b2280 [3/7389bdfb-9491-4580-9c72-b6dfcd587c3b] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:53:59.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2026-02-20T22:53:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: image deletions should propagate' 2026-02-20T22:53:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:53:59.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:53:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:53:59 TEST: image deletions should propagate 2026-02-20T22:53:59.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:53:59 TEST: image deletions should propagate 2026-02-20T22:53:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_images cluster1 mirror 0 2026-02-20T22:53:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_count=0 2026-02-20T22:53:59.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local count 2026-02-20T22:53:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local last_count=0 2026-02-20T22:53:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ true 2026-02-20T22:53:59.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:53:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:53:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:53:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:53:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'images: ' 2026-02-20T22:53:59.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 2 2026-02-20T22:53:59.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:53:59.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ count=0 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state 2026-02-20T22:53:59.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 40 2026-02-20T22:53:59.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in `seq 1 40` 2026-02-20T22:53:59.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 1 -ne 1 2026-02-20T22:53:59.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2026-02-20T22:53:59.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep 'image health:' 2026-02-20T22:53:59.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cut '-d ' -f 3 2026-02-20T22:53:59.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: failed to query services: (13) Permission denied 2026-02-20T22:53:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ state=OK 2026-02-20T22:53:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = ERROR 2026-02-20T22:53:59.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test OK = OK 2026-02-20T22:53:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 50 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_1 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:53:59.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-20T22:53:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:53:59.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:53:59.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_2 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:53:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:53:59.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-20T22:53:59.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:53:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:53:59.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.454 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-20T22:53:59.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:53:59.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:53:59.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_3 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:53:59.461 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:53:59.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:53:59.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-20T22:53:59.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:53:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2026-02-20T22:53:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:53:59.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:53:59.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-20T22:53:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:53:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:53:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_4 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:53:59.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:53:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:53:59.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-20T22:53:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2026-02-20T22:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:53:59.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-20T22:53:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:53:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:53:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_5 2026-02-20T22:53:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2026-02-20T22:53:59.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:53:59.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-20T22:53:59.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2026-02-20T22:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:53:59.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:53:59.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:53:59.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-20T22:53:59.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:53:59.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:53:59.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_6 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:53:59.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:53:59.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:53:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:53:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:53:59.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:53:59.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-20T22:53:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:53:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:53:59.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:54:00.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:54:00.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-20T22:54:00.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:00.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:00.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_7 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:54:00.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:54:00.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:54:00.067 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-20T22:54:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:00.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2026-02-20T22:54:00.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.190 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:54:00.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-20T22:54:00.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:00.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:00.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_8 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:54:00.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:54:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:54:00.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-20T22:54:00.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:00.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:00.238 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:54:00.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_9 2026-02-20T22:54:00.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:54:00.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:54:00.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.388 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-20T22:54:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:00.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:54:00.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_10 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T22:54:00.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-20T22:54:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:00.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T22:54:00.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T22:54:00.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:00.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_11 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_11 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_11 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2026-02-20T22:54:00.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:00.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_11: (2) No such file or directory 2026-02-20T22:54:00.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:00.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:00.897 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_11 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:00.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_11 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_11 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2026-02-20T22:54:01.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_11: (2) No such file or directory 2026-02-20T22:54:01.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_12 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_12 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_12 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2026-02-20T22:54:01.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_12: (2) No such file or directory 2026-02-20T22:54:01.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2026-02-20T22:54:01.151 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_12 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_12 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_12 2026-02-20T22:54:01.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2026-02-20T22:54:01.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_12: (2) No such file or directory 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_13 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2026-02-20T22:54:01.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_13 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_13 2026-02-20T22:54:01.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2026-02-20T22:54:01.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_13: (2) No such file or directory 2026-02-20T22:54:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_13 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_13 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_13 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2026-02-20T22:54:01.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_13: (2) No such file or directory 2026-02-20T22:54:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_14 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_14 2026-02-20T22:54:01.327 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_14 2026-02-20T22:54:01.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2026-02-20T22:54:01.328 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_14: (2) No such file or directory 2026-02-20T22:54:01.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.371 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7042587787160415) > weight (0.25) 2026-02-20T22:54:01.371 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8909956095611141) > weight (0.25) 2026-02-20T22:54:01.371 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2639616986708684) > weight (0.25) 2026-02-20T22:54:01.371 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6289428728231852) > weight (0.25) 2026-02-20T22:54:01.371 INFO:tasks.rbd_mirror_thrash:waiting for 63.906048957074496 secs before thrashing 2026-02-20T22:54:01.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_14 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_14 2026-02-20T22:54:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_14 2026-02-20T22:54:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2026-02-20T22:54:01.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_14: (2) No such file or directory 2026-02-20T22:54:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_15 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_15 2026-02-20T22:54:01.476 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_15 2026-02-20T22:54:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2026-02-20T22:54:01.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_15: (2) No such file or directory 2026-02-20T22:54:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2026-02-20T22:54:01.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_15 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_15 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_15 2026-02-20T22:54:01.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2026-02-20T22:54:01.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_15: (2) No such file or directory 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_16 2026-02-20T22:54:01.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_16 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_16 2026-02-20T22:54:01.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2026-02-20T22:54:01.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.645 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_16: (2) No such file or directory 2026-02-20T22:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2026-02-20T22:54:01.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_16 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_16 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_16 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2026-02-20T22:54:01.751 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_16: (2) No such file or directory 2026-02-20T22:54:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_17 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_17 2026-02-20T22:54:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_17 2026-02-20T22:54:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2026-02-20T22:54:01.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_17: (2) No such file or directory 2026-02-20T22:54:01.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_17 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:01.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_17 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_17 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2026-02-20T22:54:01.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_17: (2) No such file or directory 2026-02-20T22:54:01.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:01.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_18 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_18 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_18 2026-02-20T22:54:01.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2026-02-20T22:54:01.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:01.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_18: (2) No such file or directory 2026-02-20T22:54:01.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:01.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:01.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2026-02-20T22:54:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_18 2026-02-20T22:54:02.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_18 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_18 2026-02-20T22:54:02.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2026-02-20T22:54:02.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_18: (2) No such file or directory 2026-02-20T22:54:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_19 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_19 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_19 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2026-02-20T22:54:02.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_19: (2) No such file or directory 2026-02-20T22:54:02.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2026-02-20T22:54:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_19 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_19 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_19 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2026-02-20T22:54:02.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.225 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_19: (2) No such file or directory 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_20 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2026-02-20T22:54:02.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_20 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_20 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2026-02-20T22:54:02.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_20: (2) No such file or directory 2026-02-20T22:54:02.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_20 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.355 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_20 2026-02-20T22:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_20 2026-02-20T22:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2026-02-20T22:54:02.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_20: (2) No such file or directory 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_21 2026-02-20T22:54:02.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_21 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_21 2026-02-20T22:54:02.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2026-02-20T22:54:02.381 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_21: (2) No such file or directory 2026-02-20T22:54:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2026-02-20T22:54:02.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_21 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_21 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_21 2026-02-20T22:54:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2026-02-20T22:54:02.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_21: (2) No such file or directory 2026-02-20T22:54:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_22 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_22 2026-02-20T22:54:02.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_22 2026-02-20T22:54:02.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2026-02-20T22:54:02.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_22: (2) No such file or directory 2026-02-20T22:54:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.553 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2026-02-20T22:54:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_22 2026-02-20T22:54:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_22 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.658 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_22 2026-02-20T22:54:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2026-02-20T22:54:02.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_22: (2) No such file or directory 2026-02-20T22:54:02.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_23 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:54:02.683 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_23 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_23 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2026-02-20T22:54:02.684 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_23: (2) No such file or directory 2026-02-20T22:54:02.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2026-02-20T22:54:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_23 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_23 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_23 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2026-02-20T22:54:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.832 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_23: (2) No such file or directory 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_24 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.836 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_24 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_24 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2026-02-20T22:54:02.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_24: (2) No such file or directory 2026-02-20T22:54:02.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:02.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:02.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2026-02-20T22:54:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.963 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_24 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_24 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_24 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2026-02-20T22:54:02.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:02.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_24: (2) No such file or directory 2026-02-20T22:54:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:02.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_25 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_25 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_25 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2026-02-20T22:54:02.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_25: (2) No such file or directory 2026-02-20T22:54:03.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_25 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_25 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_25 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2026-02-20T22:54:03.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_25: (2) No such file or directory 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_26 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_26 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_26 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2026-02-20T22:54:03.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_26: (2) No such file or directory 2026-02-20T22:54:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_26 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_26 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_26 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2026-02-20T22:54:03.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_26: (2) No such file or directory 2026-02-20T22:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_27 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_27 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_27 2026-02-20T22:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2026-02-20T22:54:03.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_27: (2) No such file or directory 2026-02-20T22:54:03.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2026-02-20T22:54:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_27 2026-02-20T22:54:03.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_27 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_27 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2026-02-20T22:54:03.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_27: (2) No such file or directory 2026-02-20T22:54:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.441 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_28 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_28 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_28 2026-02-20T22:54:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2026-02-20T22:54:03.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_28: (2) No such file or directory 2026-02-20T22:54:03.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2026-02-20T22:54:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_28 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_28 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_28 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2026-02-20T22:54:03.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_28: (2) No such file or directory 2026-02-20T22:54:03.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_29 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_29 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_29 2026-02-20T22:54:03.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2026-02-20T22:54:03.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_29: (2) No such file or directory 2026-02-20T22:54:03.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2026-02-20T22:54:03.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_29 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_29 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_29 2026-02-20T22:54:03.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2026-02-20T22:54:03.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_29: (2) No such file or directory 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_30 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_30 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_30 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2026-02-20T22:54:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.762 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_30: (2) No such file or directory 2026-02-20T22:54:03.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2026-02-20T22:54:03.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_30 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_30 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_30 2026-02-20T22:54:03.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2026-02-20T22:54:03.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_30: (2) No such file or directory 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_31 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:54:03.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_31 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_31 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2026-02-20T22:54:03.895 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:03.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_31: (2) No such file or directory 2026-02-20T22:54:03.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:03.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:03.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_31 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_31 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_31 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2026-02-20T22:54:04.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_31: (2) No such file or directory 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_32 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_32 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_32 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2026-02-20T22:54:04.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_32: (2) No such file or directory 2026-02-20T22:54:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:04.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_32 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_32 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_32 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2026-02-20T22:54:04.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_32: (2) No such file or directory 2026-02-20T22:54:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_33 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_33 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_33 2026-02-20T22:54:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2026-02-20T22:54:04.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_33: (2) No such file or directory 2026-02-20T22:54:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:04.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_33 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_33 2026-02-20T22:54:04.320 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_33 2026-02-20T22:54:04.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2026-02-20T22:54:04.321 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_33: (2) No such file or directory 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_34 2026-02-20T22:54:04.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_34 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_34 2026-02-20T22:54:04.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2026-02-20T22:54:04.348 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_34: (2) No such file or directory 2026-02-20T22:54:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:04.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2026-02-20T22:54:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_34 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_34 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_34 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2026-02-20T22:54:04.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_34: (2) No such file or directory 2026-02-20T22:54:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:04.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_35 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_35 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_35 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2026-02-20T22:54:04.496 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_35: (2) No such file or directory 2026-02-20T22:54:04.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:04.520 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:04.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_35 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_35 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.623 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_35 2026-02-20T22:54:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2026-02-20T22:54:04.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_35: (2) No such file or directory 2026-02-20T22:54:04.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:04.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_36 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_36 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_36 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2026-02-20T22:54:04.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_36: (2) No such file or directory 2026-02-20T22:54:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:04.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_36 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_36 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_36 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2026-02-20T22:54:04.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_36: (2) No such file or directory 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_37 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_37 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_37 2026-02-20T22:54:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2026-02-20T22:54:05.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_37: (2) No such file or directory 2026-02-20T22:54:05.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_37 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_37 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_37 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2026-02-20T22:54:05.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_37: (2) No such file or directory 2026-02-20T22:54:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.158 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_38 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_38 2026-02-20T22:54:05.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_38 2026-02-20T22:54:05.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2026-02-20T22:54:05.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_38: (2) No such file or directory 2026-02-20T22:54:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_38 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_38 2026-02-20T22:54:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_38 2026-02-20T22:54:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2026-02-20T22:54:05.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_38: (2) No such file or directory 2026-02-20T22:54:05.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_39 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_39 2026-02-20T22:54:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_39 2026-02-20T22:54:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2026-02-20T22:54:05.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_39: (2) No such file or directory 2026-02-20T22:54:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_39 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_39 2026-02-20T22:54:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_39 2026-02-20T22:54:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2026-02-20T22:54:05.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_39: (2) No such file or directory 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_40 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:54:05.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_40 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_40 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2026-02-20T22:54:05.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_40: (2) No such file or directory 2026-02-20T22:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_40 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_40 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_40 2026-02-20T22:54:05.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2026-02-20T22:54:05.585 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.604 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_40: (2) No such file or directory 2026-02-20T22:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_41 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_41 2026-02-20T22:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_41 2026-02-20T22:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2026-02-20T22:54:05.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.629 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_41: (2) No such file or directory 2026-02-20T22:54:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2026-02-20T22:54:05.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_41 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_41 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_41 2026-02-20T22:54:05.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2026-02-20T22:54:05.736 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_41: (2) No such file or directory 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_42 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_42 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_42 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2026-02-20T22:54:05.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_42: (2) No such file or directory 2026-02-20T22:54:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:05.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2026-02-20T22:54:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_42 2026-02-20T22:54:05.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_42 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_42 2026-02-20T22:54:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2026-02-20T22:54:05.890 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_42: (2) No such file or directory 2026-02-20T22:54:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:05.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_43 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_43 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:05.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:05.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_43 2026-02-20T22:54:05.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2026-02-20T22:54:05.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:05.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_43: (2) No such file or directory 2026-02-20T22:54:05.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:05.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:05.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:06.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_43 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_43 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_43 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2026-02-20T22:54:06.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_43: (2) No such file or directory 2026-02-20T22:54:06.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:06.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_44 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_44 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_44 2026-02-20T22:54:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2026-02-20T22:54:06.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.284 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_44: (2) No such file or directory 2026-02-20T22:54:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:06.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:06.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_44 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_44 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_44 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2026-02-20T22:54:06.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_44: (2) No such file or directory 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_45 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_45 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_45 2026-02-20T22:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2026-02-20T22:54:06.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.434 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_45: (2) No such file or directory 2026-02-20T22:54:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:06.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:06.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2026-02-20T22:54:06.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_45 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_45 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_45 2026-02-20T22:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2026-02-20T22:54:06.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_45: (2) No such file or directory 2026-02-20T22:54:06.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_46 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_46 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_46 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2026-02-20T22:54:06.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_46: (2) No such file or directory 2026-02-20T22:54:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:06.591 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:06.592 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2026-02-20T22:54:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_46 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_46 2026-02-20T22:54:06.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_46 2026-02-20T22:54:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2026-02-20T22:54:06.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.716 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_46: (2) No such file or directory 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_47 2026-02-20T22:54:06.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_47 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_47 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2026-02-20T22:54:06.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_47: (2) No such file or directory 2026-02-20T22:54:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:06.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:06.846 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_47 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:06.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_47 2026-02-20T22:54:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_47 2026-02-20T22:54:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2026-02-20T22:54:06.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:06.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_47: (2) No such file or directory 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_48 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2026-02-20T22:54:06.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_48 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_48 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2026-02-20T22:54:06.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_48: (2) No such file or directory 2026-02-20T22:54:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:07.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:07.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_48 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_48 2026-02-20T22:54:07.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:07.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:07.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_48 2026-02-20T22:54:07.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2026-02-20T22:54:07.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_48: (2) No such file or directory 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_49 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_49 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_49 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2026-02-20T22:54:07.227 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_49: (2) No such file or directory 2026-02-20T22:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_49 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:07.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_49 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_49 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2026-02-20T22:54:07.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_49: (2) No such file or directory 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_50 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2026-02-20T22:54:07.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_50 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_50 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2026-02-20T22:54:07.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_50: (2) No such file or directory 2026-02-20T22:54:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_50 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:07.707 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_50 2026-02-20T22:54:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_50 2026-02-20T22:54:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2026-02-20T22:54:07.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:07.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_50: (2) No such file or directory 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: delete images during bootstrap' 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T22:54:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T22:54:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 22:54:07 TEST: delete images during bootstrap 2026-02-20T22:54:07.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 22:54:07 TEST: delete images during bootstrap 2026-02-20T22:54:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_pool_mirror_mode cluster1 mirror image 2026-02-20T22:54:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=image 2026-02-20T22:54:07.733 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2026-02-20T22:54:07.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd: mirroring is already configured for image mode 2026-02-20T22:54:07.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_pool_mirror_mode cluster2 mirror image 2026-02-20T22:54:07.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:07.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=image 2026-02-20T22:54:07.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2026-02-20T22:54:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd: mirroring is already configured for image mode 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ start_mirror cluster1 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set_cluster_instance cluster1 cluster instance 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local val=cluster1 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster_var_name=cluster 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local instance_var_name=instance 2026-02-20T22:54:07.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cluster=cluster1 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=cluster1 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' cluster1 = cluster1 ']' 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ instance=0 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval cluster=cluster1 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ cluster=cluster1 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ eval instance=0 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ instance=0 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=test 2026-02-20T22:54:07.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 1 10 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_1 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_1 512M 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:07.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:54:07.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:54:07.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:54:07.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:07.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_1 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:54:07.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2026-02-20T22:54:07.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:54:07.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2026-02-20T22:54:07.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_1': 2026-02-20T22:54:07.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1885f68d4475 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1885f68d4475 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:54:07 2026 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:54:07 2026 2026-02-20T22:54:07.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:54:07 2026 2026-02-20T22:54:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:54:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:54:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 8ac0f69e-70b7-4b32-b94e-236d6900e31a 2026-02-20T22:54:07.989 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:54:07.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_1 2026-02-20T22:54:07.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:07.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:07.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:07.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:54:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=11 2026-02-20T22:54:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:54:07.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:54:08.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 30705664 bytes 2026-02-20T22:54:08.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 105240576 bytes 2026-02-20T22:54:08.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 128231424 bytes 2026-02-20T22:54:08.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 182837248 bytes 2026-02-20T22:54:08.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 213579776 bytes 2026-02-20T22:54:08.367 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 275972096 bytes 2026-02-20T22:54:08.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 305985536 bytes 2026-02-20T22:54:08.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 307214336 bytes 2026-02-20T22:54:08.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 329134080 bytes 2026-02-20T22:54:08.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 352721920 bytes 2026-02-20T22:54:08.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 413512704 bytes 2026-02-20T22:54:08.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 434563072 bytes 2026-02-20T22:54:08.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 441256960 bytes 2026-02-20T22:54:08.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 488278016 bytes 2026-02-20T22:54:08.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 538752000 bytes 2026-02-20T22:54:08.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 638784512 bytes 2026-02-20T22:54:08.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 703412224 bytes 2026-02-20T22:54:09.049 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 776944640 bytes 2026-02-20T22:54:09.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 805599232 bytes 2026-02-20T22:54:09.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 831598592 bytes 2026-02-20T22:54:09.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 889761792 bytes 2026-02-20T22:54:09.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 962110464 bytes 2026-02-20T22:54:09.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1051463680 bytes 2026-02-20T22:54:09.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1105316864 bytes 2026-02-20T22:54:09.469 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1113208832 bytes 2026-02-20T22:54:09.532 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1169214464 bytes 2026-02-20T22:54:09.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1212177408 bytes 2026-02-20T22:54:09.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1270990848 bytes 2026-02-20T22:54:09.702 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1282471936 bytes 2026-02-20T22:54:09.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1324112896 bytes 2026-02-20T22:54:09.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1367711744 bytes 2026-02-20T22:54:09.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1415825408 bytes 2026-02-20T22:54:09.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1438507008 bytes 2026-02-20T22:54:09.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1469696000 bytes 2026-02-20T22:54:10.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1529190400 bytes 2026-02-20T22:54:10.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1564087296 bytes 2026-02-20T22:54:10.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1570865152 bytes 2026-02-20T22:54:10.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1582014464 bytes 2026-02-20T22:54:10.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1601737728 bytes 2026-02-20T22:54:10.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1653890048 bytes 2026-02-20T22:54:10.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1712620544 bytes 2026-02-20T22:54:10.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1716993024 bytes 2026-02-20T22:54:10.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1776878592 bytes 2026-02-20T22:54:10.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1816967168 bytes 2026-02-20T22:54:10.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1858243584 bytes 2026-02-20T22:54:10.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1935350784 bytes 2026-02-20T22:54:10.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1963985920 bytes 2026-02-20T22:54:10.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2032194560 bytes 2026-02-20T22:54:10.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2072954880 bytes 2026-02-20T22:54:10.849 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2122024960 bytes 2026-02-20T22:54:10.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2123991040 bytes 2026-02-20T22:54:10.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2142083072 bytes 2026-02-20T22:54:10.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2207156224 bytes 2026-02-20T22:54:11.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2268932096 bytes 2026-02-20T22:54:11.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2336432128 bytes 2026-02-20T22:54:11.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2337046528 bytes 2026-02-20T22:54:11.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2405841920 bytes 2026-02-20T22:54:11.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2449416192 bytes 2026-02-20T22:54:11.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2500359168 bytes 2026-02-20T22:54:11.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2573680640 bytes 2026-02-20T22:54:11.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2590784512 bytes 2026-02-20T22:54:11.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2633195520 bytes 2026-02-20T22:54:11.598 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2671345664 bytes 2026-02-20T22:54:11.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2757377024 bytes 2026-02-20T22:54:11.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2864675840 bytes 2026-02-20T22:54:11.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2885124096 bytes 2026-02-20T22:54:11.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2946847744 bytes 2026-02-20T22:54:11.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3021066240 bytes 2026-02-20T22:54:12.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3056260096 bytes 2026-02-20T22:54:12.114 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3143490560 bytes 2026-02-20T22:54:12.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3203514368 bytes 2026-02-20T22:54:12.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3258147840 bytes 2026-02-20T22:54:12.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3267051520 bytes 2026-02-20T22:54:12.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3276552192 bytes 2026-02-20T22:54:12.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3354632192 bytes 2026-02-20T22:54:12.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3409419264 bytes 2026-02-20T22:54:12.549 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3445580800 bytes 2026-02-20T22:54:12.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3516750848 bytes 2026-02-20T22:54:12.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3554884608 bytes 2026-02-20T22:54:12.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3571189760 bytes 2026-02-20T22:54:12.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3609207808 bytes 2026-02-20T22:54:12.822 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3655005184 bytes 2026-02-20T22:54:12.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3655626752 bytes 2026-02-20T22:54:12.894 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3656731648 bytes 2026-02-20T22:54:12.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3695818752 bytes 2026-02-20T22:54:13.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3757654016 bytes 2026-02-20T22:54:13.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3821608960 bytes 2026-02-20T22:54:13.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3876528128 bytes 2026-02-20T22:54:13.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3929146368 bytes 2026-02-20T22:54:13.283 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3975782400 bytes 2026-02-20T22:54:13.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4041602048 bytes 2026-02-20T22:54:13.406 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4085286912 bytes 2026-02-20T22:54:13.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4110144512 bytes 2026-02-20T22:54:13.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4131539968 bytes 2026-02-20T22:54:13.536 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4146515968 bytes 2026-02-20T22:54:13.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4159101952 bytes 2026-02-20T22:54:13.626 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4194490368 bytes 2026-02-20T22:54:13.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4269526016 bytes 2026-02-20T22:54:13.744 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4289529856 bytes 2026-02-20T22:54:13.786 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4305309696 bytes 2026-02-20T22:54:13.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4392379392 bytes 2026-02-20T22:54:13.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4514158592 bytes 2026-02-20T22:54:14.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4541254656 bytes 2026-02-20T22:54:14.083 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4584394752 bytes 2026-02-20T22:54:14.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4620727296 bytes 2026-02-20T22:54:14.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4680465408 bytes 2026-02-20T22:54:14.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4716185600 bytes 2026-02-20T22:54:14.280 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4730132480 bytes 2026-02-20T22:54:14.380 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4835100672 bytes 2026-02-20T22:54:14.463 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4941161472 bytes 2026-02-20T22:54:14.569 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5037450240 bytes 2026-02-20T22:54:14.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5092068352 bytes 2026-02-20T22:54:14.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5121783808 bytes 2026-02-20T22:54:14.739 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5179088896 bytes 2026-02-20T22:54:14.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5244771328 bytes 2026-02-20T22:54:14.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5310070784 bytes 2026-02-20T22:54:14.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5331841024 bytes 2026-02-20T22:54:14.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5371764736 bytes 2026-02-20T22:54:15.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5439560704 bytes 2026-02-20T22:54:15.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5476573184 bytes 2026-02-20T22:54:15.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5536807936 bytes 2026-02-20T22:54:15.219 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5580477440 bytes 2026-02-20T22:54:15.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5581106176 bytes 2026-02-20T22:54:15.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5583856640 bytes 2026-02-20T22:54:15.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5600342016 bytes 2026-02-20T22:54:15.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5618588672 bytes 2026-02-20T22:54:15.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5684986880 bytes 2026-02-20T22:54:15.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5737820160 bytes 2026-02-20T22:54:15.565 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5791738880 bytes 2026-02-20T22:54:15.615 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5818505216 bytes 2026-02-20T22:54:15.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5826979840 bytes 2026-02-20T22:54:15.686 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5827594240 bytes 2026-02-20T22:54:15.759 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5899107328 bytes 2026-02-20T22:54:15.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5928296448 bytes 2026-02-20T22:54:15.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5970949120 bytes 2026-02-20T22:54:15.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5975593984 bytes 2026-02-20T22:54:15.950 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5987633152 bytes 2026-02-20T22:54:16.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6029650944 bytes 2026-02-20T22:54:16.077 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6075295744 bytes 2026-02-20T22:54:16.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6141460480 bytes 2026-02-20T22:54:16.214 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6189002752 bytes 2026-02-20T22:54:16.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6223456256 bytes 2026-02-20T22:54:16.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6248363008 bytes 2026-02-20T22:54:16.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6284060672 bytes 2026-02-20T22:54:16.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6355023872 bytes 2026-02-20T22:54:16.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6438025216 bytes 2026-02-20T22:54:16.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6494116864 bytes 2026-02-20T22:54:16.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6540102656 bytes 2026-02-20T22:54:16.669 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6557195264 bytes 2026-02-20T22:54:16.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6569647104 bytes 2026-02-20T22:54:16.756 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6587456512 bytes 2026-02-20T22:54:16.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6663266304 bytes 2026-02-20T22:54:16.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6691997696 bytes 2026-02-20T22:54:16.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6696059904 bytes 2026-02-20T22:54:16.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6696671232 bytes 2026-02-20T22:54:16.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6714664960 bytes 2026-02-20T22:54:17.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6805968896 bytes 2026-02-20T22:54:17.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6828918784 bytes 2026-02-20T22:54:17.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6870216704 bytes 2026-02-20T22:54:17.216 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6911704064 bytes 2026-02-20T22:54:17.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6998075392 bytes 2026-02-20T22:54:17.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7020182528 bytes 2026-02-20T22:54:17.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7072742400 bytes 2026-02-20T22:54:17.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7106053120 bytes 2026-02-20T22:54:17.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7189390336 bytes 2026-02-20T22:54:17.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7199518720 bytes 2026-02-20T22:54:17.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7270202368 bytes 2026-02-20T22:54:17.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7280280576 bytes 2026-02-20T22:54:17.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7310734336 bytes 2026-02-20T22:54:17.791 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7360688128 bytes 2026-02-20T22:54:17.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7439178752 bytes 2026-02-20T22:54:17.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7522703360 bytes 2026-02-20T22:54:18.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7547167744 bytes 2026-02-20T22:54:18.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7613780992 bytes 2026-02-20T22:54:18.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7719479296 bytes 2026-02-20T22:54:18.252 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7799492608 bytes 2026-02-20T22:54:18.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7825481728 bytes 2026-02-20T22:54:18.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7844253696 bytes 2026-02-20T22:54:18.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7917775872 bytes 2026-02-20T22:54:18.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7966667776 bytes 2026-02-20T22:54:18.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7977440256 bytes 2026-02-20T22:54:18.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8090888192 bytes 2026-02-20T22:54:18.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8120058880 bytes 2026-02-20T22:54:18.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8197182464 bytes 2026-02-20T22:54:18.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8209630208 bytes 2026-02-20T22:54:18.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8210242560 bytes 2026-02-20T22:54:18.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8283259904 bytes 2026-02-20T22:54:18.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8302574592 bytes 2026-02-20T22:54:18.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:54:19.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:54:19.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12535ffa0d6c 2026-02-20T22:54:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:19.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:19.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_1 present 12535ffa0d6c 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12535ffa0d6c 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:19.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:54:19.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:19.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12535ffa0d6c 2026-02-20T22:54:19.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12535ffa0d6c 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12535ffa0d6c -o 12535ffa0d6c = 12535ffa0d6c 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_1 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:19.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2026-02-20T22:54:19.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:54:19.969+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b099dd0c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2026-02-20T22:54:19.978 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:54:19.969+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b099dd0c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=185, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2026-02-20T22:54:21.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:54:21.329+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589105696c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:54:21.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:54:21.329+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558910e65180 [3/8ac0f69e-70b7-4b32-b94e-236d6900e31a] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:54:24.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:24.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:54:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:54:24.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:25.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-20T22:54:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:54:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:54:25.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_1 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:54:25.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_1 2026-02-20T22:54:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2026-02-20T22:54:25.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:54:25.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_1: (2) No such file or directory 2026-02-20T22:54:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:54:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:54:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:54:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:25.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_1 2026-02-20T22:54:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:25.142 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2026-02-20T22:54:25.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_1 2026-02-20T22:54:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:54:25.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_1 2026-02-20T22:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_1 2026-02-20T22:54:25.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2026-02-20T22:54:25.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2026-02-20T22:54:47.758 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9957171126940213) > weight (0.25) 2026-02-20T22:54:47.758 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2026-02-20T22:54:47.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2026-02-20T22:54:47.759 INFO:tasks.rbd_mirror_thrash:waiting for 7.8345260142461495 secs before reviving daemons 2026-02-20T22:54:47.759 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T22:54:47.753+0000 7f4bd4bea640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 42643) UID: 1000 2026-02-20T22:54:49.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:54:49.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_2 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_2 512M 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:54:49.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2026-02-20T22:54:49.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_2 2026-02-20T22:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:54:49.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2026-02-20T22:54:50.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:54:50.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2026-02-20T22:54:50.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_2': 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 189a62f3d5d9 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.189a62f3d5d9 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:54:49 2026 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:54:49 2026 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:54:49 2026 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: e8abdc0b-f803-45f1-ba7b-3699f367471e 2026-02-20T22:54:50.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:54:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_2 2026-02-20T22:54:50.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:54:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:54:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:54:50.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:54:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=12 2026-02-20T22:54:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:54:50.805 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:54:50.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 29451264 bytes 2026-02-20T22:54:50.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 32436224 bytes 2026-02-20T22:54:51.001 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 129115136 bytes 2026-02-20T22:54:51.080 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 197113856 bytes 2026-02-20T22:54:51.122 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 217991168 bytes 2026-02-20T22:54:51.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 269902848 bytes 2026-02-20T22:54:51.267 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 374367232 bytes 2026-02-20T22:54:51.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 382286848 bytes 2026-02-20T22:54:51.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 422981632 bytes 2026-02-20T22:54:51.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 453969920 bytes 2026-02-20T22:54:51.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 454577152 bytes 2026-02-20T22:54:51.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 537405440 bytes 2026-02-20T22:54:51.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 558176256 bytes 2026-02-20T22:54:51.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 596028416 bytes 2026-02-20T22:54:51.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 650412032 bytes 2026-02-20T22:54:51.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 699152384 bytes 2026-02-20T22:54:51.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 740562944 bytes 2026-02-20T22:54:51.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 786653184 bytes 2026-02-20T22:54:51.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 849886208 bytes 2026-02-20T22:54:51.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 862132224 bytes 2026-02-20T22:54:51.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 895628288 bytes 2026-02-20T22:54:52.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 940681216 bytes 2026-02-20T22:54:52.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1020235776 bytes 2026-02-20T22:54:52.181 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1065478144 bytes 2026-02-20T22:54:52.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1122784256 bytes 2026-02-20T22:54:52.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1145397248 bytes 2026-02-20T22:54:52.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1185631232 bytes 2026-02-20T22:54:52.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1261686784 bytes 2026-02-20T22:54:52.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1321149440 bytes 2026-02-20T22:54:52.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1363989504 bytes 2026-02-20T22:54:52.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1389879296 bytes 2026-02-20T22:54:52.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1438476288 bytes 2026-02-20T22:54:52.715 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1484339200 bytes 2026-02-20T22:54:52.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1532147712 bytes 2026-02-20T22:54:52.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1563310080 bytes 2026-02-20T22:54:52.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1589620736 bytes 2026-02-20T22:54:52.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1649441792 bytes 2026-02-20T22:54:52.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1672010752 bytes 2026-02-20T22:54:53.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1771205632 bytes 2026-02-20T22:54:53.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1841961984 bytes 2026-02-20T22:54:53.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1884189696 bytes 2026-02-20T22:54:53.243 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1910969344 bytes 2026-02-20T22:54:53.309 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1986534400 bytes 2026-02-20T22:54:53.366 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2018272256 bytes 2026-02-20T22:54:53.400 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2022799360 bytes 2026-02-20T22:54:53.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2068627456 bytes 2026-02-20T22:54:53.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2118337536 bytes 2026-02-20T22:54:53.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2157195264 bytes 2026-02-20T22:54:53.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2210626560 bytes 2026-02-20T22:54:53.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2214752256 bytes 2026-02-20T22:54:53.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2258481152 bytes 2026-02-20T22:54:53.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2269977600 bytes 2026-02-20T22:54:53.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2333844480 bytes 2026-02-20T22:54:53.912 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2401762304 bytes 2026-02-20T22:54:53.970 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2445632512 bytes 2026-02-20T22:54:54.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2501364736 bytes 2026-02-20T22:54:54.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2565547008 bytes 2026-02-20T22:54:54.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2591374336 bytes 2026-02-20T22:54:54.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2613952512 bytes 2026-02-20T22:54:54.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2685972480 bytes 2026-02-20T22:54:54.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2716382208 bytes 2026-02-20T22:54:54.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2774153216 bytes 2026-02-20T22:54:54.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2796106752 bytes 2026-02-20T22:54:54.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2838722560 bytes 2026-02-20T22:54:54.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2895065088 bytes 2026-02-20T22:54:54.621 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2979330048 bytes 2026-02-20T22:54:54.885 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3035375616 bytes 2026-02-20T22:54:54.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3088765952 bytes 2026-02-20T22:54:55.005 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3117743104 bytes 2026-02-20T22:54:55.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3118362624 bytes 2026-02-20T22:54:55.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3146701824 bytes 2026-02-20T22:54:55.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3189951488 bytes 2026-02-20T22:54:55.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3230120960 bytes 2026-02-20T22:54:55.253 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3262876672 bytes 2026-02-20T22:54:55.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3377556480 bytes 2026-02-20T22:54:55.433 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3480293376 bytes 2026-02-20T22:54:55.477 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3480912896 bytes 2026-02-20T22:54:55.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3520309248 bytes 2026-02-20T22:54:55.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3534713856 bytes 2026-02-20T22:54:55.594 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2026-02-20T22:54:55.594 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:54:55.594 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2026-02-20T22:54:55.594 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2026-02-20T22:54:55.595 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2026-02-20T22:54:55.597 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2026-02-20T22:54:55.597 INFO:tasks.rbd_mirror_thrash:waiting for 71.50673539518505 secs before thrashing 2026-02-20T22:54:55.633 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3577188352 bytes 2026-02-20T22:54:55.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3695763456 bytes 2026-02-20T22:54:55.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3724076032 bytes 2026-02-20T22:54:55.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3762111488 bytes 2026-02-20T22:54:55.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3789355008 bytes 2026-02-20T22:54:55.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3888696320 bytes 2026-02-20T22:54:56.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3913179136 bytes 2026-02-20T22:54:56.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3967912960 bytes 2026-02-20T22:54:56.152 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3990830080 bytes 2026-02-20T22:54:56.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4025245696 bytes 2026-02-20T22:54:56.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4039532544 bytes 2026-02-20T22:54:56.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4048908288 bytes 2026-02-20T22:54:56.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4095156224 bytes 2026-02-20T22:54:56.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4145342464 bytes 2026-02-20T22:54:56.446 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4189140992 bytes 2026-02-20T22:54:56.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4248020992 bytes 2026-02-20T22:54:56.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4265546752 bytes 2026-02-20T22:54:56.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4281099264 bytes 2026-02-20T22:54:56.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4352924672 bytes 2026-02-20T22:54:56.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4367087616 bytes 2026-02-20T22:54:56.841 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4532627456 bytes 2026-02-20T22:54:56.903 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4594688000 bytes 2026-02-20T22:54:56.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4634023936 bytes 2026-02-20T22:54:57.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4646786048 bytes 2026-02-20T22:54:57.070 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4701526016 bytes 2026-02-20T22:54:57.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4822269952 bytes 2026-02-20T22:54:57.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4854547456 bytes 2026-02-20T22:54:57.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4884476928 bytes 2026-02-20T22:54:57.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4893479936 bytes 2026-02-20T22:54:57.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4950220800 bytes 2026-02-20T22:54:57.452 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5039602688 bytes 2026-02-20T22:54:57.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5058765824 bytes 2026-02-20T22:54:57.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5062211584 bytes 2026-02-20T22:54:57.577 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5092146176 bytes 2026-02-20T22:54:57.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5092783104 bytes 2026-02-20T22:54:57.657 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5136377856 bytes 2026-02-20T22:54:57.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5285707776 bytes 2026-02-20T22:54:57.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5318876160 bytes 2026-02-20T22:54:57.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5364720640 bytes 2026-02-20T22:54:57.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5365340160 bytes 2026-02-20T22:54:57.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5425356800 bytes 2026-02-20T22:54:58.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5475906560 bytes 2026-02-20T22:54:58.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5514404864 bytes 2026-02-20T22:54:58.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5515013120 bytes 2026-02-20T22:54:58.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5530178560 bytes 2026-02-20T22:54:58.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5548118016 bytes 2026-02-20T22:54:58.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5569967104 bytes 2026-02-20T22:54:58.318 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5587511296 bytes 2026-02-20T22:54:58.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5645778944 bytes 2026-02-20T22:54:58.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5698615296 bytes 2026-02-20T22:54:58.525 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5773921280 bytes 2026-02-20T22:54:58.580 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5800481792 bytes 2026-02-20T22:54:58.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5878515712 bytes 2026-02-20T22:54:58.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5933933568 bytes 2026-02-20T22:54:58.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5938345984 bytes 2026-02-20T22:54:58.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5980130304 bytes 2026-02-20T22:54:58.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6023168000 bytes 2026-02-20T22:54:58.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6087333888 bytes 2026-02-20T22:54:58.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6106199040 bytes 2026-02-20T22:54:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6184929280 bytes 2026-02-20T22:54:59.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6245398528 bytes 2026-02-20T22:54:59.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6297334784 bytes 2026-02-20T22:54:59.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6364476416 bytes 2026-02-20T22:54:59.289 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6384539648 bytes 2026-02-20T22:54:59.332 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6413293568 bytes 2026-02-20T22:54:59.399 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6456564736 bytes 2026-02-20T22:54:59.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6458017792 bytes 2026-02-20T22:54:59.481 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6493692928 bytes 2026-02-20T22:54:59.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6524237824 bytes 2026-02-20T22:54:59.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6563862528 bytes 2026-02-20T22:54:59.655 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6637589504 bytes 2026-02-20T22:54:59.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6710961152 bytes 2026-02-20T22:54:59.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6711595008 bytes 2026-02-20T22:54:59.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6752505856 bytes 2026-02-20T22:54:59.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6753155072 bytes 2026-02-20T22:54:59.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6773564416 bytes 2026-02-20T22:54:59.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6794043392 bytes 2026-02-20T22:55:00.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6845942784 bytes 2026-02-20T22:55:00.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6899899392 bytes 2026-02-20T22:55:00.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6908750848 bytes 2026-02-20T22:55:00.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6963020800 bytes 2026-02-20T22:55:00.261 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7002149888 bytes 2026-02-20T22:55:00.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7056124928 bytes 2026-02-20T22:55:00.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7084647424 bytes 2026-02-20T22:55:00.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7096162304 bytes 2026-02-20T22:55:00.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7129257984 bytes 2026-02-20T22:55:00.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7205252096 bytes 2026-02-20T22:55:00.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7279472640 bytes 2026-02-20T22:55:00.681 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7331024896 bytes 2026-02-20T22:55:00.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7339800576 bytes 2026-02-20T22:55:00.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7427524608 bytes 2026-02-20T22:55:00.858 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7470205952 bytes 2026-02-20T22:55:00.899 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7491933184 bytes 2026-02-20T22:55:00.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7516105728 bytes 2026-02-20T22:55:01.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7554420736 bytes 2026-02-20T22:55:01.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7581129728 bytes 2026-02-20T22:55:01.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7584416768 bytes 2026-02-20T22:55:01.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7604865024 bytes 2026-02-20T22:55:01.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7617683456 bytes 2026-02-20T22:55:01.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7704030208 bytes 2026-02-20T22:55:01.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7769354240 bytes 2026-02-20T22:55:01.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7858505728 bytes 2026-02-20T22:55:01.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7859144704 bytes 2026-02-20T22:55:01.491 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7890654208 bytes 2026-02-20T22:55:01.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7988550656 bytes 2026-02-20T22:55:01.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8052290560 bytes 2026-02-20T22:55:01.719 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8101128192 bytes 2026-02-20T22:55:01.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8146224128 bytes 2026-02-20T22:55:01.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8189528064 bytes 2026-02-20T22:55:01.886 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8225508352 bytes 2026-02-20T22:55:01.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8292000768 bytes 2026-02-20T22:55:02.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8349551616 bytes 2026-02-20T22:55:02.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8412118016 bytes 2026-02-20T22:55:02.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8437713920 bytes 2026-02-20T22:55:02.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8439778304 bytes 2026-02-20T22:55:02.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8456570880 bytes 2026-02-20T22:55:02.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8482854912 bytes 2026-02-20T22:55:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8539435008 bytes 2026-02-20T22:55:02.403 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8598461440 bytes 2026-02-20T22:55:02.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8674332672 bytes 2026-02-20T22:55:02.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8787682304 bytes 2026-02-20T22:55:02.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8865215488 bytes 2026-02-20T22:55:02.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8898874368 bytes 2026-02-20T22:55:02.766 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8954770432 bytes 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:55:02.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:02.811 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:55:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=1253b84334f3 2026-02-20T22:55:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:55:02.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:55:02.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_2 present 1253b84334f3 2026-02-20T22:55:02.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=1253b84334f3 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:02.947 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:55:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:55:02.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:02.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=1253b84334f3 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1253b84334f3 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 1253b84334f3 -o 1253b84334f3 = 1253b84334f3 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_2 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:02.977 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2026-02-20T22:55:03.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:55:03.785+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a963800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2026-02-20T22:55:03.791 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:55:03.785+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a963800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=186, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2026-02-20T22:55:05.278 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:55:05.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:55:05.278 INFO:tasks.rbd_mirror_thrash:waiting for 4.006188513838527 secs before reviving daemons 2026-02-20T22:55:05.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:55:05.273+0000 7fba4fd31640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 33230) UID: 1000 2026-02-20T22:55:08.795 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:55:08.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:08.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-20T22:55:08.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:55:08.830 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:55:08.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:55:08.933 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_2 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_2 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2026-02-20T22:55:08.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:08.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_2: (2) No such file or directory 2026-02-20T22:55:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:55:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:55:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:08.959 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_2 2026-02-20T22:55:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:08.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2026-02-20T22:55:08.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_2 2026-02-20T22:55:08.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:08.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:08.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:08.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:55:08.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:55:08.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_2 2026-02-20T22:55:08.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:08.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:08.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_2 2026-02-20T22:55:08.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2026-02-20T22:55:09.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2026-02-20T22:55:09.285 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:55:09.286 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:55:09.286 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:55:09.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:55:09.286 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:55:09.287 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:55:09.287 INFO:tasks.rbd_mirror_thrash:waiting for 81.12190998264249 secs before thrashing 2026-02-20T22:55:33.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_3 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_3 512M 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:33.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:55:33.286 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_3 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:55:33.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2026-02-20T22:55:33.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:55:33.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_3': 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 18c29721a0cf 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.18c29721a0cf 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:55:33 2026 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:55:33 2026 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:55:33 2026 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 87f800ad-fa6b-4a95-a563-56680a444af4 2026-02-20T22:55:34.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:55:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_3 2026-02-20T22:55:34.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:34.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:34.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:34.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:55:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:55:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:55:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:55:34.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 31801344 bytes 2026-02-20T22:55:34.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 64687104 bytes 2026-02-20T22:55:34.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 164286464 bytes 2026-02-20T22:55:34.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 200569856 bytes 2026-02-20T22:55:34.337 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 216838144 bytes 2026-02-20T22:55:34.385 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 259424256 bytes 2026-02-20T22:55:34.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 320334848 bytes 2026-02-20T22:55:34.506 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 355468288 bytes 2026-02-20T22:55:34.548 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 379694080 bytes 2026-02-20T22:55:34.601 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 427200512 bytes 2026-02-20T22:55:34.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 451911680 bytes 2026-02-20T22:55:34.689 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 474919936 bytes 2026-02-20T22:55:34.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 521943040 bytes 2026-02-20T22:55:34.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 603666432 bytes 2026-02-20T22:55:34.855 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 607746048 bytes 2026-02-20T22:55:34.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 699290624 bytes 2026-02-20T22:55:35.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 752321536 bytes 2026-02-20T22:55:35.072 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 789297152 bytes 2026-02-20T22:55:35.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 848128000 bytes 2026-02-20T22:55:35.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 933670912 bytes 2026-02-20T22:55:35.314 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1017102336 bytes 2026-02-20T22:55:35.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1044457472 bytes 2026-02-20T22:55:35.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1178095616 bytes 2026-02-20T22:55:35.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1192200192 bytes 2026-02-20T22:55:35.560 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1204118528 bytes 2026-02-20T22:55:35.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1243720704 bytes 2026-02-20T22:55:35.667 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1315523584 bytes 2026-02-20T22:55:35.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1385416704 bytes 2026-02-20T22:55:35.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1405808640 bytes 2026-02-20T22:55:35.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1482073088 bytes 2026-02-20T22:55:35.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1545124864 bytes 2026-02-20T22:55:36.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1597788160 bytes 2026-02-20T22:55:36.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1623189504 bytes 2026-02-20T22:55:36.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1673265152 bytes 2026-02-20T22:55:36.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1710692352 bytes 2026-02-20T22:55:36.247 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1800736768 bytes 2026-02-20T22:55:36.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1843897344 bytes 2026-02-20T22:55:36.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1874579456 bytes 2026-02-20T22:55:36.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1893519360 bytes 2026-02-20T22:55:36.464 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1940008960 bytes 2026-02-20T22:55:36.529 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 2001532928 bytes 2026-02-20T22:55:36.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2085650432 bytes 2026-02-20T22:55:36.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2118546432 bytes 2026-02-20T22:55:36.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2205320192 bytes 2026-02-20T22:55:36.818 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2250420224 bytes 2026-02-20T22:55:36.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2288307200 bytes 2026-02-20T22:55:36.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2309604352 bytes 2026-02-20T22:55:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2353528832 bytes 2026-02-20T22:55:37.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2369276928 bytes 2026-02-20T22:55:37.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2423972864 bytes 2026-02-20T22:55:37.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2501698560 bytes 2026-02-20T22:55:37.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2549841920 bytes 2026-02-20T22:55:37.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2638072832 bytes 2026-02-20T22:55:37.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2725971968 bytes 2026-02-20T22:55:37.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2738149376 bytes 2026-02-20T22:55:37.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2786532352 bytes 2026-02-20T22:55:37.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2894164992 bytes 2026-02-20T22:55:37.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2966301696 bytes 2026-02-20T22:55:37.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2986486784 bytes 2026-02-20T22:55:37.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 3058152448 bytes 2026-02-20T22:55:37.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 3087410176 bytes 2026-02-20T22:55:37.920 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 3187902464 bytes 2026-02-20T22:55:37.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 3246624768 bytes 2026-02-20T22:55:38.057 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 3306184704 bytes 2026-02-20T22:55:38.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3315217408 bytes 2026-02-20T22:55:38.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3326966784 bytes 2026-02-20T22:55:38.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3339513856 bytes 2026-02-20T22:55:38.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3372694528 bytes 2026-02-20T22:55:38.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3406952448 bytes 2026-02-20T22:55:38.331 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3445511168 bytes 2026-02-20T22:55:38.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3493282816 bytes 2026-02-20T22:55:38.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3519072256 bytes 2026-02-20T22:55:38.487 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3555185664 bytes 2026-02-20T22:55:38.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3601552384 bytes 2026-02-20T22:55:38.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3640753152 bytes 2026-02-20T22:55:38.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3712720896 bytes 2026-02-20T22:55:38.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3777529856 bytes 2026-02-20T22:55:38.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3792206848 bytes 2026-02-20T22:55:38.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3845510144 bytes 2026-02-20T22:55:38.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3856019456 bytes 2026-02-20T22:55:38.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3940271104 bytes 2026-02-20T22:55:39.017 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3960774656 bytes 2026-02-20T22:55:39.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3990490112 bytes 2026-02-20T22:55:39.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 4027309056 bytes 2026-02-20T22:55:39.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4056261632 bytes 2026-02-20T22:55:39.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4122771456 bytes 2026-02-20T22:55:39.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4207076352 bytes 2026-02-20T22:55:39.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4233741312 bytes 2026-02-20T22:55:39.473 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4332999680 bytes 2026-02-20T22:55:39.556 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4416484352 bytes 2026-02-20T22:55:39.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4438590464 bytes 2026-02-20T22:55:39.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4494182400 bytes 2026-02-20T22:55:39.734 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4540487680 bytes 2026-02-20T22:55:39.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4572824576 bytes 2026-02-20T22:55:39.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4646930432 bytes 2026-02-20T22:55:39.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4703300608 bytes 2026-02-20T22:55:39.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4746418176 bytes 2026-02-20T22:55:40.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4785004544 bytes 2026-02-20T22:55:40.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4814409728 bytes 2026-02-20T22:55:40.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4833241088 bytes 2026-02-20T22:55:40.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4900385792 bytes 2026-02-20T22:55:40.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4961812480 bytes 2026-02-20T22:55:40.304 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4962446336 bytes 2026-02-20T22:55:40.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4994192384 bytes 2026-02-20T22:55:40.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 5053941760 bytes 2026-02-20T22:55:40.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 5085200384 bytes 2026-02-20T22:55:40.568 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 5197053952 bytes 2026-02-20T22:55:40.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 5251375104 bytes 2026-02-20T22:55:40.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 5310674944 bytes 2026-02-20T22:55:40.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5345316864 bytes 2026-02-20T22:55:40.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5400213504 bytes 2026-02-20T22:55:40.863 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5437936640 bytes 2026-02-20T22:55:40.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5474702336 bytes 2026-02-20T22:55:40.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5483085824 bytes 2026-02-20T22:55:41.039 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5580014592 bytes 2026-02-20T22:55:41.106 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5643423744 bytes 2026-02-20T22:55:41.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5649498112 bytes 2026-02-20T22:55:41.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5693883392 bytes 2026-02-20T22:55:41.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5703956480 bytes 2026-02-20T22:55:41.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5748728832 bytes 2026-02-20T22:55:41.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5787947008 bytes 2026-02-20T22:55:41.435 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5879363584 bytes 2026-02-20T22:55:41.493 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5920629760 bytes 2026-02-20T22:55:41.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5939763200 bytes 2026-02-20T22:55:41.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5967649792 bytes 2026-02-20T22:55:41.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 6045406208 bytes 2026-02-20T22:55:41.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 6088833024 bytes 2026-02-20T22:55:41.794 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 6147333120 bytes 2026-02-20T22:55:41.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 6248388608 bytes 2026-02-20T22:55:41.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6253903872 bytes 2026-02-20T22:55:41.973 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6313654272 bytes 2026-02-20T22:55:42.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6339215360 bytes 2026-02-20T22:55:42.093 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6392297472 bytes 2026-02-20T22:55:42.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6434243584 bytes 2026-02-20T22:55:42.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6505344000 bytes 2026-02-20T22:55:42.303 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6581771264 bytes 2026-02-20T22:55:42.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6653535232 bytes 2026-02-20T22:55:42.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6707049472 bytes 2026-02-20T22:55:42.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6745724928 bytes 2026-02-20T22:55:42.599 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6838719488 bytes 2026-02-20T22:55:42.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6849026048 bytes 2026-02-20T22:55:42.682 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6864855040 bytes 2026-02-20T22:55:42.775 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6994233344 bytes 2026-02-20T22:55:42.866 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 7074528256 bytes 2026-02-20T22:55:42.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 7117362176 bytes 2026-02-20T22:55:42.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 7121358848 bytes 2026-02-20T22:55:43.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 7221449728 bytes 2026-02-20T22:55:43.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 7243521024 bytes 2026-02-20T22:55:43.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 7284002816 bytes 2026-02-20T22:55:43.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 7343862784 bytes 2026-02-20T22:55:43.285 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 7414230016 bytes 2026-02-20T22:55:43.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 7476427776 bytes 2026-02-20T22:55:43.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 7522762752 bytes 2026-02-20T22:55:43.470 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 7559135232 bytes 2026-02-20T22:55:43.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7608154112 bytes 2026-02-20T22:55:43.581 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7624860672 bytes 2026-02-20T22:55:43.637 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7686534144 bytes 2026-02-20T22:55:43.711 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7726229504 bytes 2026-02-20T22:55:43.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7800232960 bytes 2026-02-20T22:55:43.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7835828224 bytes 2026-02-20T22:55:43.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7890646016 bytes 2026-02-20T22:55:44.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7990990848 bytes 2026-02-20T22:55:44.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 8014766080 bytes 2026-02-20T22:55:44.121 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 8076900352 bytes 2026-02-20T22:55:44.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 8185836544 bytes 2026-02-20T22:55:44.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 8207362048 bytes 2026-02-20T22:55:44.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 8247529472 bytes 2026-02-20T22:55:44.370 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 8291474432 bytes 2026-02-20T22:55:44.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 8352179200 bytes 2026-02-20T22:55:44.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 8390934528 bytes 2026-02-20T22:55:44.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 8440518656 bytes 2026-02-20T22:55:44.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 8491189248 bytes 2026-02-20T22:55:44.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 8556037120 bytes 2026-02-20T22:55:44.750 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 8556655616 bytes 2026-02-20T22:55:44.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 8624950272 bytes 2026-02-20T22:55:44.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 8641871872 bytes 2026-02-20T22:55:44.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 8692649984 bytes 2026-02-20T22:55:45.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 8761191424 bytes 2026-02-20T22:55:45.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 8812354560 bytes 2026-02-20T22:55:45.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 8834789376 bytes 2026-02-20T22:55:45.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8888776704 bytes 2026-02-20T22:55:45.256 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8937032704 bytes 2026-02-20T22:55:45.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 9059857408 bytes 2026-02-20T22:55:45.429 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 9111627776 bytes 2026-02-20T22:55:45.555 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 9264756736 bytes 2026-02-20T22:55:45.624 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 9310581760 bytes 2026-02-20T22:55:45.706 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 9387231232 bytes 2026-02-20T22:55:45.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 9426025472 bytes 2026-02-20T22:55:45.819 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 9445950464 bytes 2026-02-20T22:55:45.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 9503477760 bytes 2026-02-20T22:55:45.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 9506889728 bytes 2026-02-20T22:55:45.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 9533213696 bytes 2026-02-20T22:55:46.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 9547529216 bytes 2026-02-20T22:55:46.148 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 9684445184 bytes 2026-02-20T22:55:46.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 9714163712 bytes 2026-02-20T22:55:46.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 9738159104 bytes 2026-02-20T22:55:46.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 9843160064 bytes 2026-02-20T22:55:46.417 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 9870195712 bytes 2026-02-20T22:55:46.485 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 9921048576 bytes 2026-02-20T22:55:46.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 9966898176 bytes 2026-02-20T22:55:46.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 10006845440 bytes 2026-02-20T22:55:46.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 10045322240 bytes 2026-02-20T22:55:46.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 10086291456 bytes 2026-02-20T22:55:46.793 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 10175201280 bytes 2026-02-20T22:55:46.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 10209618944 bytes 2026-02-20T22:55:46.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 10235344896 bytes 2026-02-20T22:55:46.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 10260290560 bytes 2026-02-20T22:55:47.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 10286193664 bytes 2026-02-20T22:55:47.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 10365003776 bytes 2026-02-20T22:55:47.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 10470660096 bytes 2026-02-20T22:55:47.248 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 10530541568 bytes 2026-02-20T22:55:47.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 10549174272 bytes 2026-02-20T22:55:47.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 10607466496 bytes 2026-02-20T22:55:47.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 10636297216 bytes 2026-02-20T22:55:47.467 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 10668240896 bytes 2026-02-20T22:55:47.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 10679499776 bytes 2026-02-20T22:55:47.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 10701151232 bytes 2026-02-20T22:55:47.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 10740606976 bytes 2026-02-20T22:55:47.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 10798751744 bytes 2026-02-20T22:55:47.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 10827655168 bytes 2026-02-20T22:55:47.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 10889590784 bytes 2026-02-20T22:55:47.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 10898706432 bytes 2026-02-20T22:55:47.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 10964569088 bytes 2026-02-20T22:55:48.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:55:48.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=1253727522af 2026-02-20T22:55:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:55:48.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:55:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_3 present 1253727522af 2026-02-20T22:55:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:48.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=1253727522af 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:55:48.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=1253727522af 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1253727522af 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 1253727522af -o 1253727522af = 1253727522af 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_3 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:48.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2026-02-20T22:55:48.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:55:48.980+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a9b7c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2026-02-20T22:55:48.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:55:48.980+0000 7f40a15fc640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a9b7c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2026-02-20T22:55:51.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:55:51.332+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589106a09c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:55:51.338 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:55:51.332+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e64780 [3/87f800ad-fa6b-4a95-a563-56680a444af4] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:55:53.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:53.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:55:54.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:54.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-20T22:55:54.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:55:54.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:55:54.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:55:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2026-02-20T22:55:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:55:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:54.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_3 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_3 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2026-02-20T22:55:54.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:55:54.150 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_3: (2) No such file or directory 2026-02-20T22:55:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:55:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:55:54.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_3 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:54.154 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_3 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:55:54.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:55:54.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_3 2026-02-20T22:55:54.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:55:54.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:55:54.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_3 2026-02-20T22:55:54.187 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2026-02-20T22:55:54.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.2951404548609179) > weight (0.25) 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6385454135303205) > weight (0.25) 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.5698734529318416) > weight (0.25) 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-20T22:56:07.105 INFO:tasks.rbd_mirror_thrash:waiting for 8.828169242522737 secs before reviving daemons 2026-02-20T22:56:07.106 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T22:56:07.100+0000 7f2fd8300640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 45100) UID: 1000 2026-02-20T22:56:15.934 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-20T22:56:15.934 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:56:15.935 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-20T22:56:15.935 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:56:15.935 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:56:15.937 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:56:15.937 INFO:tasks.rbd_mirror_thrash:waiting for 92.68833267194465 secs before thrashing 2026-02-20T22:56:18.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:56:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:56:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:56:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_4 2026-02-20T22:56:18.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_4 512M 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:56:18.295 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_4 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:56:18.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2026-02-20T22:56:18.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:56:18.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_4': 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 18e089dc6118 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.18e089dc6118 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:56:19.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:56:18 2026 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:56:18 2026 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:56:18 2026 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 0153fd99-6f4d-41cd-9739-5b331bb3c604 2026-02-20T22:56:19.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:56:19.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_4 2026-02-20T22:56:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:19.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:56:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=12 2026-02-20T22:56:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:56:19.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:56:19.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 40406016 bytes 2026-02-20T22:56:19.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 53632000 bytes 2026-02-20T22:56:19.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 119754752 bytes 2026-02-20T22:56:19.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 159121408 bytes 2026-02-20T22:56:19.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 171021312 bytes 2026-02-20T22:56:19.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 263336960 bytes 2026-02-20T22:56:19.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 292875264 bytes 2026-02-20T22:56:19.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 354657280 bytes 2026-02-20T22:56:19.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 400379904 bytes 2026-02-20T22:56:19.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 427524096 bytes 2026-02-20T22:56:19.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 462226432 bytes 2026-02-20T22:56:19.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 522814464 bytes 2026-02-20T22:56:19.799 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 612034560 bytes 2026-02-20T22:56:19.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 640716800 bytes 2026-02-20T22:56:19.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 681349120 bytes 2026-02-20T22:56:19.936 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 685384704 bytes 2026-02-20T22:56:20.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 760570880 bytes 2026-02-20T22:56:20.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 780853248 bytes 2026-02-20T22:56:20.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 828375040 bytes 2026-02-20T22:56:20.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 856204288 bytes 2026-02-20T22:56:20.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 924044288 bytes 2026-02-20T22:56:20.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 926224384 bytes 2026-02-20T22:56:20.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 949899264 bytes 2026-02-20T22:56:20.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 954276864 bytes 2026-02-20T22:56:20.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1001388032 bytes 2026-02-20T22:56:20.465 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1057303552 bytes 2026-02-20T22:56:20.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1099182080 bytes 2026-02-20T22:56:20.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1145832448 bytes 2026-02-20T22:56:20.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1185282048 bytes 2026-02-20T22:56:20.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1226448896 bytes 2026-02-20T22:56:20.745 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1269530624 bytes 2026-02-20T22:56:20.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1317736448 bytes 2026-02-20T22:56:20.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1369586688 bytes 2026-02-20T22:56:20.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1396762624 bytes 2026-02-20T22:56:21.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1465094144 bytes 2026-02-20T22:56:21.082 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1536591872 bytes 2026-02-20T22:56:21.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1579027456 bytes 2026-02-20T22:56:21.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1657889792 bytes 2026-02-20T22:56:21.273 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1683544064 bytes 2026-02-20T22:56:21.326 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1710090240 bytes 2026-02-20T22:56:21.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1753458688 bytes 2026-02-20T22:56:21.431 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1786547200 bytes 2026-02-20T22:56:21.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1814978560 bytes 2026-02-20T22:56:21.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1843392512 bytes 2026-02-20T22:56:21.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1881211904 bytes 2026-02-20T22:56:21.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1915282432 bytes 2026-02-20T22:56:21.708 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1967202304 bytes 2026-02-20T22:56:21.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2019971072 bytes 2026-02-20T22:56:21.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2068446208 bytes 2026-02-20T22:56:21.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2096137216 bytes 2026-02-20T22:56:21.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2156133376 bytes 2026-02-20T22:56:21.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2193107968 bytes 2026-02-20T22:56:22.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2195778560 bytes 2026-02-20T22:56:22.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2262582272 bytes 2026-02-20T22:56:22.144 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2297703424 bytes 2026-02-20T22:56:22.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2327846912 bytes 2026-02-20T22:56:22.263 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2381309952 bytes 2026-02-20T22:56:22.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2381921280 bytes 2026-02-20T22:56:22.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2403525632 bytes 2026-02-20T22:56:22.401 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2424651776 bytes 2026-02-20T22:56:22.449 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2458250240 bytes 2026-02-20T22:56:22.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2498065408 bytes 2026-02-20T22:56:22.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2567002112 bytes 2026-02-20T22:56:22.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2614922240 bytes 2026-02-20T22:56:22.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2678748160 bytes 2026-02-20T22:56:22.720 INFO:tasks.ceph.cluster2.osd.0.trial107.stderr:2026-02-20T22:56:22.716+0000 7f72ea4a6640 -1 reset not still connected to 0x55c81e750680 2026-02-20T22:56:22.768 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2734066688 bytes 2026-02-20T22:56:22.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2802140160 bytes 2026-02-20T22:56:22.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2821656576 bytes 2026-02-20T22:56:22.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2838493184 bytes 2026-02-20T22:56:22.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2918808576 bytes 2026-02-20T22:56:23.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2928911360 bytes 2026-02-20T22:56:23.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2945135616 bytes 2026-02-20T22:56:23.171 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3062576128 bytes 2026-02-20T22:56:23.255 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3141241856 bytes 2026-02-20T22:56:23.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3157374976 bytes 2026-02-20T22:56:23.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3169523712 bytes 2026-02-20T22:56:23.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3229024256 bytes 2026-02-20T22:56:23.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3251402752 bytes 2026-02-20T22:56:23.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3310403584 bytes 2026-02-20T22:56:23.552 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3320015872 bytes 2026-02-20T22:56:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3344416768 bytes 2026-02-20T22:56:23.679 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3427604480 bytes 2026-02-20T22:56:23.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3472840704 bytes 2026-02-20T22:56:23.796 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3528326144 bytes 2026-02-20T22:56:23.839 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3568360448 bytes 2026-02-20T22:56:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3600748544 bytes 2026-02-20T22:56:23.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3694520320 bytes 2026-02-20T22:56:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3744145408 bytes 2026-02-20T22:56:24.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3799033856 bytes 2026-02-20T22:56:24.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3894874112 bytes 2026-02-20T22:56:24.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3927683072 bytes 2026-02-20T22:56:24.276 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3958449152 bytes 2026-02-20T22:56:24.356 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4023321600 bytes 2026-02-20T22:56:24.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4104783872 bytes 2026-02-20T22:56:24.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4137644032 bytes 2026-02-20T22:56:24.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4245871616 bytes 2026-02-20T22:56:24.627 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4293622784 bytes 2026-02-20T22:56:24.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4358720512 bytes 2026-02-20T22:56:24.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4371858432 bytes 2026-02-20T22:56:24.787 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4397315072 bytes 2026-02-20T22:56:24.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4397915136 bytes 2026-02-20T22:56:24.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4458999808 bytes 2026-02-20T22:56:24.916 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4459585536 bytes 2026-02-20T22:56:24.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4501800960 bytes 2026-02-20T22:56:25.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4570598400 bytes 2026-02-20T22:56:25.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4617913344 bytes 2026-02-20T22:56:25.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4630743040 bytes 2026-02-20T22:56:25.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4778404864 bytes 2026-02-20T22:56:25.333 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4836372480 bytes 2026-02-20T22:56:25.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4919968768 bytes 2026-02-20T22:56:25.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4988668928 bytes 2026-02-20T22:56:25.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5014622208 bytes 2026-02-20T22:56:25.603 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5041355776 bytes 2026-02-20T22:56:25.668 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5085740032 bytes 2026-02-20T22:56:25.722 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5120131072 bytes 2026-02-20T22:56:25.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5157038080 bytes 2026-02-20T22:56:25.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5203430400 bytes 2026-02-20T22:56:25.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5246007296 bytes 2026-02-20T22:56:25.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5305313280 bytes 2026-02-20T22:56:25.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5338731520 bytes 2026-02-20T22:56:26.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5364624384 bytes 2026-02-20T22:56:26.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5387956224 bytes 2026-02-20T22:56:26.145 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5412782080 bytes 2026-02-20T22:56:26.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5443890176 bytes 2026-02-20T22:56:26.240 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5472722944 bytes 2026-02-20T22:56:26.299 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5516724224 bytes 2026-02-20T22:56:26.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5533012992 bytes 2026-02-20T22:56:26.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5558868992 bytes 2026-02-20T22:56:26.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5603484672 bytes 2026-02-20T22:56:26.530 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5669665792 bytes 2026-02-20T22:56:26.582 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5702710272 bytes 2026-02-20T22:56:26.622 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5723401216 bytes 2026-02-20T22:56:26.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5734733824 bytes 2026-02-20T22:56:26.725 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5762065408 bytes 2026-02-20T22:56:26.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5794764800 bytes 2026-02-20T22:56:26.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5797465088 bytes 2026-02-20T22:56:26.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5841828864 bytes 2026-02-20T22:56:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5917404160 bytes 2026-02-20T22:56:27.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5989306368 bytes 2026-02-20T22:56:27.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6019144704 bytes 2026-02-20T22:56:27.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6039336960 bytes 2026-02-20T22:56:27.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6061186048 bytes 2026-02-20T22:56:27.211 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6108920832 bytes 2026-02-20T22:56:27.269 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6130283520 bytes 2026-02-20T22:56:27.330 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6188283904 bytes 2026-02-20T22:56:27.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6203573248 bytes 2026-02-20T22:56:27.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6253244416 bytes 2026-02-20T22:56:27.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6276301824 bytes 2026-02-20T22:56:27.524 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6303798272 bytes 2026-02-20T22:56:27.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6334209024 bytes 2026-02-20T22:56:27.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6344494080 bytes 2026-02-20T22:56:27.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6371402752 bytes 2026-02-20T22:56:27.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6431568896 bytes 2026-02-20T22:56:27.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6459384832 bytes 2026-02-20T22:56:27.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6496478208 bytes 2026-02-20T22:56:27.888 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6538093568 bytes 2026-02-20T22:56:27.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6567667712 bytes 2026-02-20T22:56:27.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6605761536 bytes 2026-02-20T22:56:28.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6633162752 bytes 2026-02-20T22:56:28.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6687305728 bytes 2026-02-20T22:56:28.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6755009536 bytes 2026-02-20T22:56:28.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6803539968 bytes 2026-02-20T22:56:28.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6867152896 bytes 2026-02-20T22:56:28.361 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6913936384 bytes 2026-02-20T22:56:28.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6925072384 bytes 2026-02-20T22:56:28.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 6929391616 bytes 2026-02-20T22:56:28.523 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7002692608 bytes 2026-02-20T22:56:28.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7036082176 bytes 2026-02-20T22:56:28.607 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7036691456 bytes 2026-02-20T22:56:28.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7060147200 bytes 2026-02-20T22:56:28.690 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7081638912 bytes 2026-02-20T22:56:28.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7107116032 bytes 2026-02-20T22:56:28.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7125051392 bytes 2026-02-20T22:56:28.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7158350848 bytes 2026-02-20T22:56:28.882 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7179787264 bytes 2026-02-20T22:56:28.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7219038208 bytes 2026-02-20T22:56:28.994 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7256961024 bytes 2026-02-20T22:56:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7290523648 bytes 2026-02-20T22:56:29.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7351023616 bytes 2026-02-20T22:56:29.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7452548096 bytes 2026-02-20T22:56:29.232 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 7453154304 bytes 2026-02-20T22:56:29.272 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 7474157568 bytes 2026-02-20T22:56:29.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 7496398848 bytes 2026-02-20T22:56:29.365 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 7525820416 bytes 2026-02-20T22:56:29.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 7548835840 bytes 2026-02-20T22:56:29.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 7607268352 bytes 2026-02-20T22:56:29.527 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 7649600512 bytes 2026-02-20T22:56:29.587 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 7697937408 bytes 2026-02-20T22:56:29.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 7732793344 bytes 2026-02-20T22:56:29.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 7744866304 bytes 2026-02-20T22:56:29.732 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 7784906752 bytes 2026-02-20T22:56:29.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 7868177408 bytes 2026-02-20T22:56:29.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 7891681280 bytes 2026-02-20T22:56:29.928 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 7943212032 bytes 2026-02-20T22:56:29.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 7970563072 bytes 2026-02-20T22:56:30.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8018947072 bytes 2026-02-20T22:56:30.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8089297920 bytes 2026-02-20T22:56:30.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8142064640 bytes 2026-02-20T22:56:30.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8162214912 bytes 2026-02-20T22:56:30.259 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8201172992 bytes 2026-02-20T22:56:30.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8227501056 bytes 2026-02-20T22:56:30.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8269930496 bytes 2026-02-20T22:56:30.410 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2026-02-20T22:56:30.410 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2026-02-20T22:56:30.410 INFO:tasks.rbd_mirror_thrash:waiting for 5.439477996593379 secs before reviving daemons 2026-02-20T22:56:30.411 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T22:56:30.408+0000 7fb55cb36640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 52790) UID: 1000 2026-02-20T22:56:30.432 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8321188864 bytes 2026-02-20T22:56:30.479 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 8352167936 bytes 2026-02-20T22:56:30.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 8409858048 bytes 2026-02-20T22:56:30.608 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 8463838208 bytes 2026-02-20T22:56:30.677 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 8533254144 bytes 2026-02-20T22:56:30.763 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 8631124992 bytes 2026-02-20T22:56:30.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 8686412800 bytes 2026-02-20T22:56:30.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 8775653376 bytes 2026-02-20T22:56:30.964 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 8833055744 bytes 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:31.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:56:31.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:56:31.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=1253c661ae78 2026-02-20T22:56:31.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:56:31.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_4 present 1253c661ae78 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:56:31.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=1253c661ae78 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:56:31.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=1253c661ae78 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1253c661ae78 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 1253c661ae78 -o 1253c661ae78 = 1253c661ae78 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:56:31.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_4 2026-02-20T22:56:31.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:31.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:31.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:31.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2026-02-20T22:56:31.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:56:31.772+0000 7f40a0dfb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b099331400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2026-02-20T22:56:31.776 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:56:31.772+0000 7f40a0dfb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b099331400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2026-02-20T22:56:35.850 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2026-02-20T22:56:35.850 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:56:35.851 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2026-02-20T22:56:35.851 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2026-02-20T22:56:35.851 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2026-02-20T22:56:35.852 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2026-02-20T22:56:35.852 INFO:tasks.rbd_mirror_thrash:waiting for 88.25422283745053 secs before thrashing 2026-02-20T22:56:36.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:56:36.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2026-02-20T22:56:36.788 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:56:36.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:56:36.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:56:36.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:56:36.810 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-20T22:56:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:56:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:56:36.813 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:56:36.914 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_4 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_4 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2026-02-20T22:56:36.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:56:36.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_4: (2) No such file or directory 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_4 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:36.941 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_4 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:56:36.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_4 2026-02-20T22:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_4 2026-02-20T22:56:36.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2026-02-20T22:56:37.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2026-02-20T22:57:00.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_5 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_5 512M 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:57:00.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:57:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:57:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:00.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2026-02-20T22:57:00.124 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_5 2026-02-20T22:57:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:57:00.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2026-02-20T22:57:00.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:57:00.986 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2026-02-20T22:57:01.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_5': 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 1902a1cb46b1 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.1902a1cb46b1 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:57:01.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:57:00 2026 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:57:00 2026 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:57:00 2026 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: cacf3a95-9b41-417c-acbd-5f3602869bf1 2026-02-20T22:57:01.013 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:57:01.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_5 2026-02-20T22:57:01.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:01.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:01.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:01.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:57:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=14 2026-02-20T22:57:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:57:01.016 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:57:01.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 51236864 bytes 2026-02-20T22:57:01.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 129793024 bytes 2026-02-20T22:57:01.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 167506944 bytes 2026-02-20T22:57:01.308 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 214315008 bytes 2026-02-20T22:57:01.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 260784128 bytes 2026-02-20T22:57:01.410 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 294956032 bytes 2026-02-20T22:57:01.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 399260672 bytes 2026-02-20T22:57:01.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 428356608 bytes 2026-02-20T22:57:01.595 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 456792064 bytes 2026-02-20T22:57:01.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 485548032 bytes 2026-02-20T22:57:01.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 563222528 bytes 2026-02-20T22:57:01.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 613897216 bytes 2026-02-20T22:57:01.831 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 633169920 bytes 2026-02-20T22:57:01.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 705079296 bytes 2026-02-20T22:57:01.952 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 747228160 bytes 2026-02-20T22:57:02.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 778352640 bytes 2026-02-20T22:57:02.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 782424064 bytes 2026-02-20T22:57:02.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 851546112 bytes 2026-02-20T22:57:02.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 872357888 bytes 2026-02-20T22:57:02.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 958377984 bytes 2026-02-20T22:57:02.298 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 989523968 bytes 2026-02-20T22:57:02.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1011180544 bytes 2026-02-20T22:57:02.447 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1143630848 bytes 2026-02-20T22:57:02.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1149816832 bytes 2026-02-20T22:57:02.515 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1152985088 bytes 2026-02-20T22:57:02.564 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1183233024 bytes 2026-02-20T22:57:02.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1215923200 bytes 2026-02-20T22:57:02.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1224130560 bytes 2026-02-20T22:57:02.714 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1283044352 bytes 2026-02-20T22:57:02.770 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1321759744 bytes 2026-02-20T22:57:02.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1326297088 bytes 2026-02-20T22:57:02.884 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1398823936 bytes 2026-02-20T22:57:02.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1464115200 bytes 2026-02-20T22:57:02.996 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1483836416 bytes 2026-02-20T22:57:03.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1506115584 bytes 2026-02-20T22:57:03.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1578914816 bytes 2026-02-20T22:57:03.182 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1647094784 bytes 2026-02-20T22:57:03.234 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1647731712 bytes 2026-02-20T22:57:03.302 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1702809600 bytes 2026-02-20T22:57:03.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1753644032 bytes 2026-02-20T22:57:03.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1789242368 bytes 2026-02-20T22:57:03.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1828203520 bytes 2026-02-20T22:57:03.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1856608256 bytes 2026-02-20T22:57:03.628 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1947419648 bytes 2026-02-20T22:57:03.696 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1995554816 bytes 2026-02-20T22:57:03.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2013158400 bytes 2026-02-20T22:57:03.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2111960064 bytes 2026-02-20T22:57:03.893 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2156790784 bytes 2026-02-20T22:57:03.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2174841856 bytes 2026-02-20T22:57:03.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2220987392 bytes 2026-02-20T22:57:04.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2241236992 bytes 2026-02-20T22:57:04.105 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2302040064 bytes 2026-02-20T22:57:04.183 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2370463744 bytes 2026-02-20T22:57:04.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2404567040 bytes 2026-02-20T22:57:04.294 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2446270464 bytes 2026-02-20T22:57:04.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2518690816 bytes 2026-02-20T22:57:04.456 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2620617728 bytes 2026-02-20T22:57:04.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2656086016 bytes 2026-02-20T22:57:04.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2731817984 bytes 2026-02-20T22:57:04.674 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2808273920 bytes 2026-02-20T22:57:04.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2864414720 bytes 2026-02-20T22:57:04.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2898767872 bytes 2026-02-20T22:57:04.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2956236800 bytes 2026-02-20T22:57:04.919 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2994541568 bytes 2026-02-20T22:57:04.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3030671360 bytes 2026-02-20T22:57:05.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3105811456 bytes 2026-02-20T22:57:05.176 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3239460864 bytes 2026-02-20T22:57:05.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3252115456 bytes 2026-02-20T22:57:05.279 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3281805312 bytes 2026-02-20T22:57:05.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3285204992 bytes 2026-02-20T22:57:05.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3332928512 bytes 2026-02-20T22:57:05.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3395734528 bytes 2026-02-20T22:57:05.539 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3476802560 bytes 2026-02-20T22:57:05.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3568711680 bytes 2026-02-20T22:57:05.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3569316864 bytes 2026-02-20T22:57:05.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3680156672 bytes 2026-02-20T22:57:05.785 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3682429952 bytes 2026-02-20T22:57:05.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3743797248 bytes 2026-02-20T22:57:05.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3793656832 bytes 2026-02-20T22:57:06.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3898295296 bytes 2026-02-20T22:57:06.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3898904576 bytes 2026-02-20T22:57:06.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3922982912 bytes 2026-02-20T22:57:06.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3972470784 bytes 2026-02-20T22:57:06.218 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 4003460096 bytes 2026-02-20T22:57:06.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 4092737536 bytes 2026-02-20T22:57:06.353 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 4134300672 bytes 2026-02-20T22:57:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4189000704 bytes 2026-02-20T22:57:06.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4203817984 bytes 2026-02-20T22:57:06.505 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4238966784 bytes 2026-02-20T22:57:06.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4309942272 bytes 2026-02-20T22:57:06.617 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4310554624 bytes 2026-02-20T22:57:06.661 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4333434880 bytes 2026-02-20T22:57:06.700 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4349647872 bytes 2026-02-20T22:57:06.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4391881728 bytes 2026-02-20T22:57:06.808 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4429415424 bytes 2026-02-20T22:57:06.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4451106816 bytes 2026-02-20T22:57:06.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4491976704 bytes 2026-02-20T22:57:06.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4543647744 bytes 2026-02-20T22:57:07.061 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4617077760 bytes 2026-02-20T22:57:07.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4688402432 bytes 2026-02-20T22:57:07.249 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4767847424 bytes 2026-02-20T22:57:07.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4785307648 bytes 2026-02-20T22:57:07.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4812110848 bytes 2026-02-20T22:57:07.383 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4822878208 bytes 2026-02-20T22:57:07.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4856902656 bytes 2026-02-20T22:57:07.484 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4879017984 bytes 2026-02-20T22:57:07.559 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4925451264 bytes 2026-02-20T22:57:07.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4954361856 bytes 2026-02-20T22:57:07.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4979120128 bytes 2026-02-20T22:57:07.731 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 5037459456 bytes 2026-02-20T22:57:07.783 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 5061522432 bytes 2026-02-20T22:57:07.848 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 5106385920 bytes 2026-02-20T22:57:07.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 5136000000 bytes 2026-02-20T22:57:07.938 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 5160974336 bytes 2026-02-20T22:57:08.055 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 5289961472 bytes 2026-02-20T22:57:08.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 5344779264 bytes 2026-02-20T22:57:08.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5374255104 bytes 2026-02-20T22:57:08.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5401096192 bytes 2026-02-20T22:57:08.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5445750784 bytes 2026-02-20T22:57:08.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5520779264 bytes 2026-02-20T22:57:08.420 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5564190720 bytes 2026-02-20T22:57:08.474 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5599754240 bytes 2026-02-20T22:57:08.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5640601600 bytes 2026-02-20T22:57:08.639 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5746563072 bytes 2026-02-20T22:57:08.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5790419968 bytes 2026-02-20T22:57:08.767 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5886927872 bytes 2026-02-20T22:57:08.838 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5907783680 bytes 2026-02-20T22:57:08.875 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5923152896 bytes 2026-02-20T22:57:08.951 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5980056576 bytes 2026-02-20T22:57:09.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 6018606080 bytes 2026-02-20T22:57:09.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 6050812928 bytes 2026-02-20T22:57:09.115 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 6127284224 bytes 2026-02-20T22:57:09.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 6155527168 bytes 2026-02-20T22:57:09.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 6204607488 bytes 2026-02-20T22:57:09.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 6213281792 bytes 2026-02-20T22:57:09.335 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6256583680 bytes 2026-02-20T22:57:09.395 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6292903936 bytes 2026-02-20T22:57:09.475 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6362984448 bytes 2026-02-20T22:57:09.545 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6425650176 bytes 2026-02-20T22:57:09.653 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6533554176 bytes 2026-02-20T22:57:09.728 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6600972288 bytes 2026-02-20T22:57:09.777 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6628958208 bytes 2026-02-20T22:57:09.840 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6671987712 bytes 2026-02-20T22:57:09.871 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6672603136 bytes 2026-02-20T22:57:09.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6717309952 bytes 2026-02-20T22:57:09.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6765300736 bytes 2026-02-20T22:57:10.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6779855872 bytes 2026-02-20T22:57:10.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6833044480 bytes 2026-02-20T22:57:10.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6918114304 bytes 2026-02-20T22:57:10.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6948461568 bytes 2026-02-20T22:57:10.277 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6983271424 bytes 2026-02-20T22:57:10.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6986773504 bytes 2026-02-20T22:57:10.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 7032444928 bytes 2026-02-20T22:57:10.415 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 7063520256 bytes 2026-02-20T22:57:10.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 7081236480 bytes 2026-02-20T22:57:10.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7124563968 bytes 2026-02-20T22:57:10.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7125185536 bytes 2026-02-20T22:57:10.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7165196288 bytes 2026-02-20T22:57:10.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7171517440 bytes 2026-02-20T22:57:10.726 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7265295360 bytes 2026-02-20T22:57:10.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7300317184 bytes 2026-02-20T22:57:10.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7349214208 bytes 2026-02-20T22:57:10.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7440864256 bytes 2026-02-20T22:57:10.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7500192768 bytes 2026-02-20T22:57:11.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7563406336 bytes 2026-02-20T22:57:11.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7577335808 bytes 2026-02-20T22:57:11.131 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7577964544 bytes 2026-02-20T22:57:11.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7596732416 bytes 2026-02-20T22:57:11.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7643086848 bytes 2026-02-20T22:57:11.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7652103168 bytes 2026-02-20T22:57:11.336 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7715208192 bytes 2026-02-20T22:57:11.423 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7754103808 bytes 2026-02-20T22:57:11.489 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7790474240 bytes 2026-02-20T22:57:11.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7791499264 bytes 2026-02-20T22:57:11.574 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7825720320 bytes 2026-02-20T22:57:11.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7873870848 bytes 2026-02-20T22:57:11.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7935977472 bytes 2026-02-20T22:57:11.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7942250496 bytes 2026-02-20T22:57:11.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7976249344 bytes 2026-02-20T22:57:11.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 8035132416 bytes 2026-02-20T22:57:11.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8101692416 bytes 2026-02-20T22:57:12.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8158460928 bytes 2026-02-20T22:57:12.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8194926592 bytes 2026-02-20T22:57:12.132 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8229009408 bytes 2026-02-20T22:57:12.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18500 IOs, 8235081728 bytes 2026-02-20T22:57:12.221 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18600 IOs, 8273283072 bytes 2026-02-20T22:57:12.293 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18700 IOs, 8321929216 bytes 2026-02-20T22:57:12.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18800 IOs, 8344957952 bytes 2026-02-20T22:57:12.393 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18900 IOs, 8385662976 bytes 2026-02-20T22:57:12.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19000 IOs, 8414406656 bytes 2026-02-20T22:57:12.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19100 IOs, 8463535104 bytes 2026-02-20T22:57:12.554 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19200 IOs, 8506340352 bytes 2026-02-20T22:57:12.634 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19300 IOs, 8570246144 bytes 2026-02-20T22:57:12.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19400 IOs, 8624723968 bytes 2026-02-20T22:57:12.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19500 IOs, 8659563520 bytes 2026-02-20T22:57:12.807 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19600 IOs, 8686795776 bytes 2026-02-20T22:57:12.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19700 IOs, 8734042112 bytes 2026-02-20T22:57:12.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19800 IOs, 8784811008 bytes 2026-02-20T22:57:12.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:19900 IOs, 8806690816 bytes 2026-02-20T22:57:13.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20000 IOs, 8833014784 bytes 2026-02-20T22:57:13.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20100 IOs, 8904313856 bytes 2026-02-20T22:57:13.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20200 IOs, 8953190400 bytes 2026-02-20T22:57:13.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20300 IOs, 8985857024 bytes 2026-02-20T22:57:13.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20400 IOs, 9078159360 bytes 2026-02-20T22:57:13.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20500 IOs, 9112071168 bytes 2026-02-20T22:57:13.412 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20600 IOs, 9139908608 bytes 2026-02-20T22:57:13.455 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20700 IOs, 9167693824 bytes 2026-02-20T22:57:13.517 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20800 IOs, 9202159616 bytes 2026-02-20T22:57:13.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:20900 IOs, 9244344320 bytes 2026-02-20T22:57:13.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21000 IOs, 9334972416 bytes 2026-02-20T22:57:13.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21100 IOs, 9364645888 bytes 2026-02-20T22:57:13.834 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21200 IOs, 9489038336 bytes 2026-02-20T22:57:13.873 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21300 IOs, 9499036672 bytes 2026-02-20T22:57:13.935 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21400 IOs, 9547419648 bytes 2026-02-20T22:57:13.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21500 IOs, 9557280768 bytes 2026-02-20T22:57:14.058 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21600 IOs, 9628337152 bytes 2026-02-20T22:57:14.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21700 IOs, 9678643200 bytes 2026-02-20T22:57:14.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21800 IOs, 9727766528 bytes 2026-02-20T22:57:14.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:21900 IOs, 9758312448 bytes 2026-02-20T22:57:14.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22000 IOs, 9771977728 bytes 2026-02-20T22:57:14.351 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22100 IOs, 9772612608 bytes 2026-02-20T22:57:14.428 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22200 IOs, 9832192000 bytes 2026-02-20T22:57:14.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22300 IOs, 9853638656 bytes 2026-02-20T22:57:14.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22400 IOs, 9908450304 bytes 2026-02-20T22:57:14.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22500 IOs, 9955969024 bytes 2026-02-20T22:57:14.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22600 IOs, 10019608576 bytes 2026-02-20T22:57:14.752 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22700 IOs, 10056941568 bytes 2026-02-20T22:57:14.821 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22800 IOs, 10106637312 bytes 2026-02-20T22:57:14.874 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:22900 IOs, 10137444352 bytes 2026-02-20T22:57:14.940 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23000 IOs, 10207448064 bytes 2026-02-20T22:57:15.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:23100 IOs, 10235352064 bytes 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:15.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:57:15.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:15.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=1253dd1ca079 2026-02-20T22:57:15.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:57:15.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:57:15.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_5 present 1253dd1ca079 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=1253dd1ca079 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:57:15.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:57:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:57:15.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=1253dd1ca079 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1253dd1ca079 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 1253dd1ca079 -o 1253dd1ca079 = 1253dd1ca079 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_5 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:15.185 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:15.186 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2026-02-20T22:57:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:57:15.959+0000 7f40a0dfb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a9b7400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2026-02-20T22:57:15.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:57:15.959+0000 7f40a0dfb640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x55b09a9b7400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2026-02-20T22:57:20.967 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:20.974 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:57:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:20.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:57:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:57:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:57:20.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:20.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-20T22:57:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:57:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:57:21.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:57:21.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_5 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_5 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2026-02-20T22:57:21.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:21.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_5: (2) No such file or directory 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_5 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:21.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_5 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:57:21.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:57:21.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_5 2026-02-20T22:57:21.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:21.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:21.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_5 2026-02-20T22:57:21.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2026-02-20T22:57:21.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2026-02-20T22:57:45.090 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:57:45.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_6 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_6 512M 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:45.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:57:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2026-02-20T22:57:45.126 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_6 2026-02-20T22:57:45.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:45.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:45.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:45.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:57:45.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2026-02-20T22:57:45.987 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:57:45.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_6': 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 191a5d70939f 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.191a5d70939f 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:57:45 2026 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:57:45 2026 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:57:45 2026 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 7dfe0ff4-b807-4ed9-8455-12744ef5f1f2 2026-02-20T22:57:46.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:57:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_6 2026-02-20T22:57:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:46.021 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:46.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:57:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=5 2026-02-20T22:57:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:57:46.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:57:46.110 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 28170240 bytes 2026-02-20T22:57:46.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 158867456 bytes 2026-02-20T22:57:46.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 159486976 bytes 2026-02-20T22:57:46.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 230374400 bytes 2026-02-20T22:57:46.357 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 232497152 bytes 2026-02-20T22:57:46.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 273889280 bytes 2026-02-20T22:57:46.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 352422912 bytes 2026-02-20T22:57:46.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 360700928 bytes 2026-02-20T22:57:46.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 434769920 bytes 2026-02-20T22:57:46.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 486703104 bytes 2026-02-20T22:57:46.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 502021120 bytes 2026-02-20T22:57:46.738 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 520801280 bytes 2026-02-20T22:57:46.820 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 616595456 bytes 2026-02-20T22:57:46.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 672517120 bytes 2026-02-20T22:57:46.942 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 717219840 bytes 2026-02-20T22:57:47.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 797504512 bytes 2026-02-20T22:57:47.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 827815936 bytes 2026-02-20T22:57:47.123 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 859367424 bytes 2026-02-20T22:57:47.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 871730176 bytes 2026-02-20T22:57:47.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 878216192 bytes 2026-02-20T22:57:47.245 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 911099904 bytes 2026-02-20T22:57:47.323 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 965620736 bytes 2026-02-20T22:57:47.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1131667456 bytes 2026-02-20T22:57:47.499 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1166570496 bytes 2026-02-20T22:57:47.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1236691968 bytes 2026-02-20T22:57:47.609 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1265465344 bytes 2026-02-20T22:57:47.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1347853312 bytes 2026-02-20T22:57:47.761 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1411305472 bytes 2026-02-20T22:57:47.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1460352000 bytes 2026-02-20T22:57:47.878 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1521966080 bytes 2026-02-20T22:57:47.924 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1543030784 bytes 2026-02-20T22:57:47.978 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1587856384 bytes 2026-02-20T22:57:48.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1609018368 bytes 2026-02-20T22:57:48.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1684468736 bytes 2026-02-20T22:57:48.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1735208960 bytes 2026-02-20T22:57:48.217 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1775496192 bytes 2026-02-20T22:57:48.254 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1785203712 bytes 2026-02-20T22:57:48.300 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1817101312 bytes 2026-02-20T22:57:48.382 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1898107904 bytes 2026-02-20T22:57:48.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1938021376 bytes 2026-02-20T22:57:48.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1981389824 bytes 2026-02-20T22:57:48.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2066401280 bytes 2026-02-20T22:57:48.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2079772672 bytes 2026-02-20T22:57:48.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.722015989039993) > weight (0.25) 2026-02-20T22:57:48.626 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.4882486320509022) > weight (0.25) 2026-02-20T22:57:48.627 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2026-02-20T22:57:48.627 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2026-02-20T22:57:48.627 INFO:tasks.rbd_mirror_thrash:waiting for 7.887005672999744 secs before reviving daemons 2026-02-20T22:57:48.628 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T22:57:48.623+0000 7f40aee17640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 30076) UID: 1000 2026-02-20T22:57:48.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2099603456 bytes 2026-02-20T22:57:48.703 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2139650048 bytes 2026-02-20T22:57:48.753 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2158767104 bytes 2026-02-20T22:57:48.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2175308800 bytes 2026-02-20T22:57:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2202092544 bytes 2026-02-20T22:57:48.900 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2233073664 bytes 2026-02-20T22:57:48.961 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2293356544 bytes 2026-02-20T22:57:49.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2305912832 bytes 2026-02-20T22:57:49.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2316492800 bytes 2026-02-20T22:57:49.096 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2350539776 bytes 2026-02-20T22:57:49.133 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2355709952 bytes 2026-02-20T22:57:49.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2403093504 bytes 2026-02-20T22:57:49.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2436005888 bytes 2026-02-20T22:57:49.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2480706560 bytes 2026-02-20T22:57:49.384 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2551707648 bytes 2026-02-20T22:57:49.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2611001344 bytes 2026-02-20T22:57:49.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2616668160 bytes 2026-02-20T22:57:49.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2665264128 bytes 2026-02-20T22:57:49.573 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2685502464 bytes 2026-02-20T22:57:49.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2740706304 bytes 2026-02-20T22:57:49.680 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2744538112 bytes 2026-02-20T22:57:49.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2753733632 bytes 2026-02-20T22:57:49.779 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2814042112 bytes 2026-02-20T22:57:49.828 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2833972224 bytes 2026-02-20T22:57:49.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2943101952 bytes 2026-02-20T22:57:49.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2958612480 bytes 2026-02-20T22:57:50.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3017391104 bytes 2026-02-20T22:57:50.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3083015168 bytes 2026-02-20T22:57:50.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3160119296 bytes 2026-02-20T22:57:50.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3214112768 bytes 2026-02-20T22:57:50.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3309437952 bytes 2026-02-20T22:57:50.346 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3320612864 bytes 2026-02-20T22:57:50.398 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3355062272 bytes 2026-02-20T22:57:50.436 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3363099648 bytes 2026-02-20T22:57:50.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3389010944 bytes 2026-02-20T22:57:50.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3428199424 bytes 2026-02-20T22:57:50.602 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3475513344 bytes 2026-02-20T22:57:50.652 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3516454912 bytes 2026-02-20T22:57:50.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3587555328 bytes 2026-02-20T22:57:50.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3642310656 bytes 2026-02-20T22:57:50.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3663728640 bytes 2026-02-20T22:57:50.913 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3708698624 bytes 2026-02-20T22:57:50.993 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3795945472 bytes 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:51.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:57:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:57:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:51.030 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:57:51.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:51.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:51.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:57:51.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:57:51.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:51.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12538224205b 2026-02-20T22:57:51.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:57:51.060 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_6 present 12538224205b 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12538224205b 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:57:51.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12538224205b 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12538224205b 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12538224205b -o 12538224205b = 12538224205b 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_6 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:51.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2026-02-20T22:57:52.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:57:52.859+0000 7f8ee2554640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x558910e34b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:57:52.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:57:52.859+0000 7f8ee9562640 -1 rbd::mirror::ImageReplayer: 0x558910e64780 [3/7dfe0ff4-b807-4ed9-8455-12744ef5f1f2] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:57:56.515 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2026-02-20T22:57:56.515 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:57:56.515 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2026-02-20T22:57:56.515 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2026-02-20T22:57:56.515 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2026-02-20T22:57:56.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2026-02-20T22:57:56.517 INFO:tasks.rbd_mirror_thrash:waiting for 83.96224355493521 secs before thrashing 2026-02-20T22:57:57.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:57:57.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:57:57.012 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:57.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-20T22:57:57.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:57:57.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:57:57.038 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2026-02-20T22:57:57.139 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_6 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_6 2026-02-20T22:57:57.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2026-02-20T22:57:57.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:57:57.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_6: (2) No such file or directory 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_6 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:57.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2026-02-20T22:57:57.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_6 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_6 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_6 2026-02-20T22:57:57.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2026-02-20T22:57:57.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2026-02-20T22:58:04.107 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7164269808658054) > weight (0.25) 2026-02-20T22:58:04.108 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.27779620686611795) > weight (0.25) 2026-02-20T22:58:04.108 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2026-02-20T22:58:04.108 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2026-02-20T22:58:04.108 INFO:tasks.rbd_mirror_thrash:waiting for 7.7581531200431595 secs before reviving daemons 2026-02-20T22:58:04.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T22:58:04.102+0000 7fc45a6fa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 46871) UID: 1000 2026-02-20T22:58:11.868 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2026-02-20T22:58:11.868 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:58:11.868 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2026-02-20T22:58:11.868 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2026-02-20T22:58:11.868 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2026-02-20T22:58:11.870 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2026-02-20T22:58:11.870 INFO:tasks.rbd_mirror_thrash:waiting for 104.4818881868939 secs before thrashing 2026-02-20T22:58:22.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:58:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:58:22.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_7 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_7 512M 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:58:22.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:58:22.099 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_7 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:58:22.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2026-02-20T22:58:22.991 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:58:23.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_7': 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 193e92a7252 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.193e92a7252 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:58:22 2026 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:58:22 2026 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:58:22 2026 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: bbb1b22a-2c3b-4d3c-9fe3-3a5567b2cfb7 2026-02-20T22:58:23.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_7 2026-02-20T22:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:23.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:58:23.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=10 2026-02-20T22:58:23.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:58:23.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:58:23.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 36336640 bytes 2026-02-20T22:58:23.168 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 58614784 bytes 2026-02-20T22:58:23.213 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 79725568 bytes 2026-02-20T22:58:23.268 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 144642048 bytes 2026-02-20T22:58:23.316 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 173570048 bytes 2026-02-20T22:58:23.389 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 231343104 bytes 2026-02-20T22:58:23.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 263221248 bytes 2026-02-20T22:58:23.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 279298048 bytes 2026-02-20T22:58:23.533 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 308209664 bytes 2026-02-20T22:58:23.571 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 312026112 bytes 2026-02-20T22:58:23.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 389915648 bytes 2026-02-20T22:58:23.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 416324608 bytes 2026-02-20T22:58:23.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 435153920 bytes 2026-02-20T22:58:23.797 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 481220608 bytes 2026-02-20T22:58:23.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 530626560 bytes 2026-02-20T22:58:23.902 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 550483968 bytes 2026-02-20T22:58:23.945 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 576281600 bytes 2026-02-20T22:58:24.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 654422016 bytes 2026-02-20T22:58:24.087 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 706324480 bytes 2026-02-20T22:58:24.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 760688640 bytes 2026-02-20T22:58:24.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 787124224 bytes 2026-02-20T22:58:24.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 787750912 bytes 2026-02-20T22:58:24.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 806009856 bytes 2026-02-20T22:58:24.341 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 858976256 bytes 2026-02-20T22:58:24.411 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 918662144 bytes 2026-02-20T22:58:24.511 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1026715648 bytes 2026-02-20T22:58:24.590 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1086471168 bytes 2026-02-20T22:58:24.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1122441216 bytes 2026-02-20T22:58:24.691 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1152655360 bytes 2026-02-20T22:58:24.747 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1182873600 bytes 2026-02-20T22:58:24.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1232738304 bytes 2026-02-20T22:58:24.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1276936192 bytes 2026-02-20T22:58:24.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1298637824 bytes 2026-02-20T22:58:24.976 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1321192448 bytes 2026-02-20T22:58:25.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1373454336 bytes 2026-02-20T22:58:25.091 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1404872704 bytes 2026-02-20T22:58:25.155 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1452052480 bytes 2026-02-20T22:58:25.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1503611904 bytes 2026-02-20T22:58:25.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1504208896 bytes 2026-02-20T22:58:25.324 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1540899840 bytes 2026-02-20T22:58:25.363 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1553837056 bytes 2026-02-20T22:58:25.430 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1607924736 bytes 2026-02-20T22:58:25.501 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1680434176 bytes 2026-02-20T22:58:25.551 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1709300736 bytes 2026-02-20T22:58:25.613 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1754090496 bytes 2026-02-20T22:58:25.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 1754713088 bytes 2026-02-20T22:58:25.692 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 1789155328 bytes 2026-02-20T22:58:25.746 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 1805220864 bytes 2026-02-20T22:58:25.809 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 1845634048 bytes 2026-02-20T22:58:25.862 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 1869745152 bytes 2026-02-20T22:58:25.915 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 1887460352 bytes 2026-02-20T22:58:25.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 1961674752 bytes 2026-02-20T22:58:26.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2028200960 bytes 2026-02-20T22:58:26.120 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2049692672 bytes 2026-02-20T22:58:26.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2067615744 bytes 2026-02-20T22:58:26.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2078686208 bytes 2026-02-20T22:58:26.296 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2175087616 bytes 2026-02-20T22:58:26.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2188021760 bytes 2026-02-20T22:58:26.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2218371072 bytes 2026-02-20T22:58:26.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2310238208 bytes 2026-02-20T22:58:26.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2319199232 bytes 2026-02-20T22:58:26.562 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2319805440 bytes 2026-02-20T22:58:26.632 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2383390720 bytes 2026-02-20T22:58:26.721 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2476962816 bytes 2026-02-20T22:58:26.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2517970944 bytes 2026-02-20T22:58:26.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2627731456 bytes 2026-02-20T22:58:26.937 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2660413440 bytes 2026-02-20T22:58:26.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2697521152 bytes 2026-02-20T22:58:27.053 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 2738911232 bytes 2026-02-20T22:58:27.117 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 2788321280 bytes 2026-02-20T22:58:27.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 2805184512 bytes 2026-02-20T22:58:27.237 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 2847662080 bytes 2026-02-20T22:58:27.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 2873021440 bytes 2026-02-20T22:58:27.342 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 2922964992 bytes 2026-02-20T22:58:27.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 2960569344 bytes 2026-02-20T22:58:27.480 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3013374976 bytes 2026-02-20T22:58:27.586 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3126463488 bytes 2026-02-20T22:58:27.638 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3150931968 bytes 2026-02-20T22:58:27.688 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3189611520 bytes 2026-02-20T22:58:27.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3294527488 bytes 2026-02-20T22:58:27.881 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3404608512 bytes 2026-02-20T22:58:27.932 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3418756096 bytes 2026-02-20T22:58:27.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3443082240 bytes 2026-02-20T22:58:28.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3470381056 bytes 2026-02-20T22:58:28.102 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3525357568 bytes 2026-02-20T22:58:28.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3571857408 bytes 2026-02-20T22:58:28.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3595704320 bytes 2026-02-20T22:58:28.282 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3660184576 bytes 2026-02-20T22:58:28.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3712336896 bytes 2026-02-20T22:58:28.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3722761216 bytes 2026-02-20T22:58:28.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3769000960 bytes 2026-02-20T22:58:28.492 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3788870656 bytes 2026-02-20T22:58:28.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3818773504 bytes 2026-02-20T22:58:28.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3834816512 bytes 2026-02-20T22:58:28.651 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3887892480 bytes 2026-02-20T22:58:28.705 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3929806848 bytes 2026-02-20T22:58:28.765 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 3969611776 bytes 2026-02-20T22:58:28.833 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4031328256 bytes 2026-02-20T22:58:28.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4121987072 bytes 2026-02-20T22:58:28.997 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4183706624 bytes 2026-02-20T22:58:29.048 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4227658752 bytes 2026-02-20T22:58:29.136 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4313444352 bytes 2026-02-20T22:58:29.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4334252032 bytes 2026-02-20T22:58:29.270 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4353410048 bytes 2026-02-20T22:58:29.339 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4413706240 bytes 2026-02-20T22:58:29.390 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4453515264 bytes 2026-02-20T22:58:29.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4515425280 bytes 2026-02-20T22:58:29.547 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4576746496 bytes 2026-02-20T22:58:29.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4581172224 bytes 2026-02-20T22:58:29.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4624629760 bytes 2026-02-20T22:58:29.698 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4646273024 bytes 2026-02-20T22:58:29.757 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4687913984 bytes 2026-02-20T22:58:29.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4716902400 bytes 2026-02-20T22:58:29.856 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4744192000 bytes 2026-02-20T22:58:29.909 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4744823808 bytes 2026-02-20T22:58:29.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4765052928 bytes 2026-02-20T22:58:30.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 4791569408 bytes 2026-02-20T22:58:30.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 4853284864 bytes 2026-02-20T22:58:30.149 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 4916800512 bytes 2026-02-20T22:58:30.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 4966671360 bytes 2026-02-20T22:58:30.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5039168512 bytes 2026-02-20T22:58:30.347 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5056352256 bytes 2026-02-20T22:58:30.387 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5075676160 bytes 2026-02-20T22:58:30.459 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5126081536 bytes 2026-02-20T22:58:30.519 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5159847936 bytes 2026-02-20T22:58:30.579 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5194429440 bytes 2026-02-20T22:58:30.641 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5243450368 bytes 2026-02-20T22:58:30.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5304426496 bytes 2026-02-20T22:58:30.776 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5352263680 bytes 2026-02-20T22:58:30.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5383282688 bytes 2026-02-20T22:58:30.879 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5418524672 bytes 2026-02-20T22:58:30.969 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5495572480 bytes 2026-02-20T22:58:31.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5498125312 bytes 2026-02-20T22:58:31.064 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5547237376 bytes 2026-02-20T22:58:31.116 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5561158656 bytes 2026-02-20T22:58:31.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 5667781632 bytes 2026-02-20T22:58:31.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 5720214528 bytes 2026-02-20T22:58:31.345 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 5756877824 bytes 2026-02-20T22:58:31.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 5778222080 bytes 2026-02-20T22:58:31.445 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 5800275968 bytes 2026-02-20T22:58:31.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 5835314176 bytes 2026-02-20T22:58:31.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 5846856704 bytes 2026-02-20T22:58:31.654 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 5926573056 bytes 2026-02-20T22:58:31.712 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 5959011328 bytes 2026-02-20T22:58:31.758 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 5974445056 bytes 2026-02-20T22:58:31.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6047419392 bytes 2026-02-20T22:58:31.949 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6150729728 bytes 2026-02-20T22:58:32.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6223215616 bytes 2026-02-20T22:58:32.088 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6271286272 bytes 2026-02-20T22:58:32.157 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6347952128 bytes 2026-02-20T22:58:32.212 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6367019008 bytes 2026-02-20T22:58:32.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6399158272 bytes 2026-02-20T22:58:32.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6399866880 bytes 2026-02-20T22:58:32.379 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6428941312 bytes 2026-02-20T22:58:32.448 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6480166912 bytes 2026-02-20T22:58:32.518 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 6537874432 bytes 2026-02-20T22:58:32.578 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 6564890624 bytes 2026-02-20T22:58:32.659 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 6632960000 bytes 2026-02-20T22:58:32.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 6677919744 bytes 2026-02-20T22:58:32.780 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 6705215488 bytes 2026-02-20T22:58:32.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 6732720128 bytes 2026-02-20T22:58:32.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 6779906048 bytes 2026-02-20T22:58:32.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 6784245760 bytes 2026-02-20T22:58:32.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 6815803392 bytes 2026-02-20T22:58:33.029 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 6839939072 bytes 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:58:33.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:58:33.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:58:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12b1a5292bde 2026-02-20T22:58:33.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:58:33.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_7 present 12b1a5292bde 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12b1a5292bde 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:58:33.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:58:33.167 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12b1a5292bde 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12b1a5292bde 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12b1a5292bde -o 12b1a5292bde = 12b1a5292bde 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_7 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:33.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2026-02-20T22:58:33.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:58:33.782+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619826bd800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2026-02-20T22:58:33.787 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:58:33.782+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619826bd800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2026-02-20T22:58:38.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:58:38.800 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:58:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:58:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:58:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:58:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:58:38.801 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:58:38.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:58:38.823 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-20T22:58:38.826 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:58:38.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:58:38.827 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:58:38.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2026-02-20T22:58:38.929 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:58:38.930 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_7 2026-02-20T22:58:38.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:58:38.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:58:38.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_7 2026-02-20T22:58:38.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2026-02-20T22:58:38.931 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:58:39.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_7: (2) No such file or directory 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_7 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:39.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:39.161 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2026-02-20T22:58:39.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_7 2026-02-20T22:58:39.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:39.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:39.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:39.191 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:58:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:58:39.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_7 2026-02-20T22:58:39.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:58:39.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:58:39.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_7 2026-02-20T22:58:39.193 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2026-02-20T22:58:39.222 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2026-02-20T22:59:02.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_8 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_8 512M 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:02.103 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:02.104 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2026-02-20T22:59:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_8 2026-02-20T22:59:02.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:02.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:02.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:02.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:59:02.135 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2026-02-20T22:59:02.990 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:59:02.998 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2026-02-20T22:59:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_8': 2026-02-20T22:59:03.022 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 194e7c90dbb 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.194e7c90dbb 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:59:03.023 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:59:02 2026 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:59:02 2026 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:59:02 2026 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: edc0a6f9-8d01-4426-8480-39ee6422bf2b 2026-02-20T22:59:03.024 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:59:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_8 2026-02-20T22:59:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:03.025 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:03.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:03.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:59:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=7 2026-02-20T22:59:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:59:03.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:59:03.095 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 3620864 bytes 2026-02-20T22:59:03.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 94042112 bytes 2026-02-20T22:59:03.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 191115264 bytes 2026-02-20T22:59:03.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 210371584 bytes 2026-02-20T22:59:03.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 244124672 bytes 2026-02-20T22:59:03.413 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 288834560 bytes 2026-02-20T22:59:03.497 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 364243968 bytes 2026-02-20T22:59:03.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 444759040 bytes 2026-02-20T22:59:03.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 479637504 bytes 2026-02-20T22:59:03.678 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 506699776 bytes 2026-02-20T22:59:03.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 560044032 bytes 2026-02-20T22:59:03.835 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 685236224 bytes 2026-02-20T22:59:03.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 768820224 bytes 2026-02-20T22:59:03.975 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 822637568 bytes 2026-02-20T22:59:04.040 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 872077312 bytes 2026-02-20T22:59:04.081 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 889704448 bytes 2026-02-20T22:59:04.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 936404992 bytes 2026-02-20T22:59:04.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 960822272 bytes 2026-02-20T22:59:04.223 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 961424384 bytes 2026-02-20T22:59:04.287 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1020642304 bytes 2026-02-20T22:59:04.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1036445696 bytes 2026-02-20T22:59:04.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1044845568 bytes 2026-02-20T22:59:04.425 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1076852736 bytes 2026-02-20T22:59:04.483 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1119125504 bytes 2026-02-20T22:59:04.526 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1133955072 bytes 2026-02-20T22:59:04.588 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1186903040 bytes 2026-02-20T22:59:04.644 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1230398464 bytes 2026-02-20T22:59:04.697 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1274989568 bytes 2026-02-20T22:59:04.742 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1294582784 bytes 2026-02-20T22:59:04.790 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1330056192 bytes 2026-02-20T22:59:04.860 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1398635520 bytes 2026-02-20T22:59:04.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1426964480 bytes 2026-02-20T22:59:04.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1441823744 bytes 2026-02-20T22:59:04.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1478626304 bytes 2026-02-20T22:59:05.065 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1522503680 bytes 2026-02-20T22:59:05.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1585714176 bytes 2026-02-20T22:59:05.170 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1592802304 bytes 2026-02-20T22:59:05.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1607021568 bytes 2026-02-20T22:59:05.260 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1639563264 bytes 2026-02-20T22:59:05.319 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1690754048 bytes 2026-02-20T22:59:05.372 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1736435712 bytes 2026-02-20T22:59:05.444 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1784629248 bytes 2026-02-20T22:59:05.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1870100480 bytes 2026-02-20T22:59:05.575 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 1893949440 bytes 2026-02-20T22:59:05.656 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 1979982848 bytes 2026-02-20T22:59:05.713 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2005470208 bytes 2026-02-20T22:59:05.772 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2046991360 bytes 2026-02-20T22:59:05.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2121013248 bytes 2026-02-20T22:59:05.907 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2154901504 bytes 2026-02-20T22:59:05.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2193653760 bytes 2026-02-20T22:59:06.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2218686464 bytes 2026-02-20T22:59:06.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2219329536 bytes 2026-02-20T22:59:06.078 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2219954176 bytes 2026-02-20T22:59:06.127 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2268277760 bytes 2026-02-20T22:59:06.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2351306752 bytes 2026-02-20T22:59:06.281 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2418208768 bytes 2026-02-20T22:59:06.338 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2483077120 bytes 2026-02-20T22:59:06.396 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2528509952 bytes 2026-02-20T22:59:06.472 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2592854016 bytes 2026-02-20T22:59:06.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2665664512 bytes 2026-02-20T22:59:06.584 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2666277888 bytes 2026-02-20T22:59:06.625 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2682300416 bytes 2026-02-20T22:59:06.676 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2702985216 bytes 2026-02-20T22:59:06.740 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2756558848 bytes 2026-02-20T22:59:06.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2809611264 bytes 2026-02-20T22:59:06.864 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2857093120 bytes 2026-02-20T22:59:06.917 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2892133376 bytes 2026-02-20T22:59:06.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 2971643904 bytes 2026-02-20T22:59:07.062 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3047744512 bytes 2026-02-20T22:59:07.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3056244736 bytes 2026-02-20T22:59:07.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3110368256 bytes 2026-02-20T22:59:07.229 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3168000000 bytes 2026-02-20T22:59:07.274 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3190703104 bytes 2026-02-20T22:59:07.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3262185472 bytes 2026-02-20T22:59:07.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3314686976 bytes 2026-02-20T22:59:07.462 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3355262976 bytes 2026-02-20T22:59:07.513 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3390515200 bytes 2026-02-20T22:59:07.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3492250624 bytes 2026-02-20T22:59:07.650 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3522585600 bytes 2026-02-20T22:59:07.718 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3573116928 bytes 2026-02-20T22:59:07.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3621702656 bytes 2026-02-20T22:59:07.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3660999680 bytes 2026-02-20T22:59:07.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3694462976 bytes 2026-02-20T22:59:07.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3714862080 bytes 2026-02-20T22:59:07.995 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3760687104 bytes 2026-02-20T22:59:08.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3790480384 bytes 2026-02-20T22:59:08.113 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3835658240 bytes 2026-02-20T22:59:08.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3838823424 bytes 2026-02-20T22:59:08.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3890786304 bytes 2026-02-20T22:59:08.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3915220992 bytes 2026-02-20T22:59:08.290 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3926217728 bytes 2026-02-20T22:59:08.352 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3982426112 bytes 2026-02-20T22:59:08.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4031848448 bytes 2026-02-20T22:59:08.498 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4121351168 bytes 2026-02-20T22:59:08.537 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4135990272 bytes 2026-02-20T22:59:08.583 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4169289728 bytes 2026-02-20T22:59:08.648 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4228193280 bytes 2026-02-20T22:59:08.737 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4333430784 bytes 2026-02-20T22:59:08.815 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4410174464 bytes 2026-02-20T22:59:08.857 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4429242368 bytes 2026-02-20T22:59:08.891 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4432269312 bytes 2026-02-20T22:59:08.962 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4504511488 bytes 2026-02-20T22:59:09.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4532814848 bytes 2026-02-20T22:59:09.056 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4573777920 bytes 2026-02-20T22:59:09.119 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4597082112 bytes 2026-02-20T22:59:09.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4601085952 bytes 2026-02-20T22:59:09.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4610109440 bytes 2026-02-20T22:59:09.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4634951680 bytes 2026-02-20T22:59:09.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4669070336 bytes 2026-02-20T22:59:09.354 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4734635008 bytes 2026-02-20T22:59:09.407 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4775965696 bytes 2026-02-20T22:59:09.458 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4824128512 bytes 2026-02-20T22:59:09.538 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4890953728 bytes 2026-02-20T22:59:09.605 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4959975424 bytes 2026-02-20T22:59:09.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4967992320 bytes 2026-02-20T22:59:09.704 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4987535360 bytes 2026-02-20T22:59:09.760 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5043275776 bytes 2026-02-20T22:59:09.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5124094976 bytes 2026-02-20T22:59:09.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5157221376 bytes 2026-02-20T22:59:09.972 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5220465664 bytes 2026-02-20T22:59:10.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:59:10.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:59:10.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:10.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12b16b8c6ce4 2026-02-20T22:59:10.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:59:10.063 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:59:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_8 present 12b16b8c6ce4 2026-02-20T22:59:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:10.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12b16b8c6ce4 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:10.165 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:59:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:59:10.166 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12b16b8c6ce4 2026-02-20T22:59:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12b16b8c6ce4 2026-02-20T22:59:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12b16b8c6ce4 -o 12b16b8c6ce4 = 12b16b8c6ce4 2026-02-20T22:59:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:59:10.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_8 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:10.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2026-02-20T22:59:11.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:59:11.002+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561981637c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2026-02-20T22:59:11.009 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:59:11.002+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x561981637c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2026-02-20T22:59:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:59:15.361+0000 7f8ee2d55640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5589116424e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2026-02-20T22:59:15.369 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T22:59:15.361+0000 7f8ee8d61640 -1 rbd::mirror::ImageReplayer: 0x558910e64c80 [3/edc0a6f9-8d01-4426-8480-39ee6422bf2b] operator(): start failed: (85) Interrupted system call should be restarted 2026-02-20T22:59:16.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:59:16.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2026-02-20T22:59:16.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:16.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:16.018 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:59:16.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:16.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-20T22:59:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:59:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:59:16.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:59:16.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_8 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_8 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2026-02-20T22:59:16.147 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:16.169 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_8: (2) No such file or directory 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_8 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:16.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2026-02-20T22:59:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_8 2026-02-20T22:59:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:16.206 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:59:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:59:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_8 2026-02-20T22:59:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:16.207 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_8 2026-02-20T22:59:16.208 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2026-02-20T22:59:16.239 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2026-02-20T22:59:20.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.9406738800819479) > weight (0.25) 2026-02-20T22:59:20.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9529991250227153) > weight (0.25) 2026-02-20T22:59:20.480 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.745951386625436) > weight (0.25) 2026-02-20T22:59:20.481 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2026-02-20T22:59:20.481 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2026-02-20T22:59:20.482 INFO:tasks.rbd_mirror_thrash:waiting for 8.801716782969589 secs before reviving daemons 2026-02-20T22:59:20.482 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T22:59:20.477+0000 7f39d99ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 53136) UID: 1000 2026-02-20T22:59:29.284 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2026-02-20T22:59:29.285 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T22:59:29.285 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2026-02-20T22:59:29.285 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2026-02-20T22:59:29.285 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2026-02-20T22:59:29.287 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2026-02-20T22:59:29.287 INFO:tasks.rbd_mirror_thrash:waiting for 79.20228179387742 secs before thrashing 2026-02-20T22:59:39.092 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_9 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_9 512M 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:39.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T22:59:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T22:59:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T22:59:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T22:59:39.098 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2026-02-20T22:59:39.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_9 2026-02-20T22:59:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T22:59:39.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2026-02-20T22:59:39.999 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T22:59:40.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_9': 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 19666a826568 2026-02-20T22:59:40.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.19666a826568 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 22:59:39 2026 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 22:59:39 2026 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 22:59:39 2026 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 1cdd7208-4818-4cac-90fe-b45ec9f62ed1 2026-02-20T22:59:40.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T22:59:40.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_9 2026-02-20T22:59:40.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:40.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:40.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:40.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T22:59:40.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=11 2026-02-20T22:59:40.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T22:59:40.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2026-02-20T22:59:40.118 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 25334784 bytes 2026-02-20T22:59:40.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 44668928 bytes 2026-02-20T22:59:40.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 45285376 bytes 2026-02-20T22:59:40.265 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 117982208 bytes 2026-02-20T22:59:40.307 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 148215808 bytes 2026-02-20T22:59:40.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 204355584 bytes 2026-02-20T22:59:40.437 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 274325504 bytes 2026-02-20T22:59:40.502 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 323879936 bytes 2026-02-20T22:59:40.558 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 354397184 bytes 2026-02-20T22:59:40.616 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 409744384 bytes 2026-02-20T22:59:40.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 434970624 bytes 2026-02-20T22:59:40.727 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 487300096 bytes 2026-02-20T22:59:40.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 588785664 bytes 2026-02-20T22:59:40.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 685745152 bytes 2026-02-20T22:59:40.946 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 710919168 bytes 2026-02-20T22:59:40.988 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 738700288 bytes 2026-02-20T22:59:41.037 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 758447104 bytes 2026-02-20T22:59:41.075 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 767909888 bytes 2026-02-20T22:59:41.129 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 803877888 bytes 2026-02-20T22:59:41.200 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 863026176 bytes 2026-02-20T22:59:41.257 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 905503744 bytes 2026-02-20T22:59:41.364 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1017108480 bytes 2026-02-20T22:59:41.440 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1085319168 bytes 2026-02-20T22:59:41.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1131940864 bytes 2026-02-20T22:59:41.541 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1153000448 bytes 2026-02-20T22:59:41.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1194967040 bytes 2026-02-20T22:59:41.663 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1256364032 bytes 2026-02-20T22:59:41.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1276040192 bytes 2026-02-20T22:59:41.782 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1352870912 bytes 2026-02-20T22:59:41.854 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1429127168 bytes 2026-02-20T22:59:41.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1434469376 bytes 2026-02-20T22:59:41.992 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1564280832 bytes 2026-02-20T22:59:42.054 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1623416832 bytes 2026-02-20T22:59:42.112 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1670838272 bytes 2026-02-20T22:59:42.153 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1684128768 bytes 2026-02-20T22:59:42.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1733305344 bytes 2026-02-20T22:59:42.271 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1766534144 bytes 2026-02-20T22:59:42.310 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1782430720 bytes 2026-02-20T22:59:42.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1834155008 bytes 2026-02-20T22:59:42.421 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1870824448 bytes 2026-02-20T22:59:42.471 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1903065088 bytes 2026-02-20T22:59:42.521 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 1930332160 bytes 2026-02-20T22:59:42.563 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 1952184320 bytes 2026-02-20T22:59:42.643 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2028518400 bytes 2026-02-20T22:59:42.694 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2058860544 bytes 2026-02-20T22:59:42.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2081310720 bytes 2026-02-20T22:59:42.781 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2094320640 bytes 2026-02-20T22:59:42.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2115750912 bytes 2026-02-20T22:59:42.889 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2166162432 bytes 2026-02-20T22:59:42.971 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2222110720 bytes 2026-02-20T22:59:43.014 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2237874176 bytes 2026-02-20T22:59:43.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2271319040 bytes 2026-02-20T22:59:43.100 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2282101760 bytes 2026-02-20T22:59:43.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2337512448 bytes 2026-02-20T22:59:43.203 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2352896000 bytes 2026-02-20T22:59:43.262 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2394774528 bytes 2026-02-20T22:59:43.317 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2435390464 bytes 2026-02-20T22:59:43.362 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2463512576 bytes 2026-02-20T22:59:43.416 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2505127936 bytes 2026-02-20T22:59:43.490 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2592452608 bytes 2026-02-20T22:59:43.550 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2635003904 bytes 2026-02-20T22:59:43.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2692213760 bytes 2026-02-20T22:59:43.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2738027520 bytes 2026-02-20T22:59:43.720 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2759725056 bytes 2026-02-20T22:59:43.774 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 2799955968 bytes 2026-02-20T22:59:43.843 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 2863912960 bytes 2026-02-20T22:59:43.944 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 2949190656 bytes 2026-02-20T22:59:44.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3000688640 bytes 2026-02-20T22:59:44.059 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3020940288 bytes 2026-02-20T22:59:44.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3026313216 bytes 2026-02-20T22:59:44.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3092741120 bytes 2026-02-20T22:59:44.205 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3113982976 bytes 2026-02-20T22:59:44.266 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3141371904 bytes 2026-02-20T22:59:44.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3184950272 bytes 2026-02-20T22:59:44.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3185561600 bytes 2026-02-20T22:59:44.397 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3201281024 bytes 2026-02-20T22:59:44.439 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3217665024 bytes 2026-02-20T22:59:44.507 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3277244416 bytes 2026-02-20T22:59:44.540 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3277836288 bytes 2026-02-20T22:59:44.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3283805184 bytes 2026-02-20T22:59:44.662 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3384921088 bytes 2026-02-20T22:59:44.710 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3418053632 bytes 2026-02-20T22:59:44.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3459606528 bytes 2026-02-20T22:59:44.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3491431424 bytes 2026-02-20T22:59:44.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3516827648 bytes 2026-02-20T22:59:44.910 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3527336960 bytes 2026-02-20T22:59:45.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 3630292992 bytes 2026-02-20T22:59:45.047 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 3680786432 bytes 2026-02-20T22:59:45.097 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 3686118400 bytes 2026-02-20T22:59:45.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 3696849920 bytes 2026-02-20T22:59:45.174 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 3713382400 bytes 2026-02-20T22:59:45.246 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 3763999744 bytes 2026-02-20T22:59:45.278 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 3764613120 bytes 2026-02-20T22:59:45.329 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 3821000704 bytes 2026-02-20T22:59:45.409 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 3874720768 bytes 2026-02-20T22:59:45.482 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 3941046272 bytes 2026-02-20T22:59:45.567 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9700 IOs, 4025397248 bytes 2026-02-20T22:59:45.619 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9800 IOs, 4061060096 bytes 2026-02-20T22:59:45.724 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9900 IOs, 4178610176 bytes 2026-02-20T22:59:45.802 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10000 IOs, 4247262208 bytes 2026-02-20T22:59:45.859 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10100 IOs, 4293961728 bytes 2026-02-20T22:59:45.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10200 IOs, 4343316480 bytes 2026-02-20T22:59:45.968 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10300 IOs, 4370896896 bytes 2026-02-20T22:59:46.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10400 IOs, 4386946048 bytes 2026-02-20T22:59:46.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10500 IOs, 4441272320 bytes 2026-02-20T22:59:46.128 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10600 IOs, 4503716864 bytes 2026-02-20T22:59:46.164 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10700 IOs, 4504364032 bytes 2026-02-20T22:59:46.204 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10800 IOs, 4518393856 bytes 2026-02-20T22:59:46.251 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:10900 IOs, 4545091584 bytes 2026-02-20T22:59:46.322 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11000 IOs, 4617334784 bytes 2026-02-20T22:59:46.443 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11100 IOs, 4754512896 bytes 2026-02-20T22:59:46.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11200 IOs, 4783349760 bytes 2026-02-20T22:59:46.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11300 IOs, 4828960768 bytes 2026-02-20T22:59:46.610 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11400 IOs, 4895520768 bytes 2026-02-20T22:59:46.687 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11500 IOs, 4963724288 bytes 2026-02-20T22:59:46.723 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11600 IOs, 4970673152 bytes 2026-02-20T22:59:46.814 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11700 IOs, 5092855808 bytes 2026-02-20T22:59:46.880 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11800 IOs, 5117178880 bytes 2026-02-20T22:59:46.922 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:11900 IOs, 5139972096 bytes 2026-02-20T22:59:46.958 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12000 IOs, 5147084800 bytes 2026-02-20T22:59:47.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12100 IOs, 5175464960 bytes 2026-02-20T22:59:47.079 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12200 IOs, 5262229504 bytes 2026-02-20T22:59:47.134 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12300 IOs, 5298230272 bytes 2026-02-20T22:59:47.189 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12400 IOs, 5353534464 bytes 2026-02-20T22:59:47.291 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12500 IOs, 5448621056 bytes 2026-02-20T22:59:47.375 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12600 IOs, 5552640000 bytes 2026-02-20T22:59:47.438 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12700 IOs, 5599756288 bytes 2026-02-20T22:59:47.494 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12800 IOs, 5631099904 bytes 2026-02-20T22:59:47.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:12900 IOs, 5644206080 bytes 2026-02-20T22:59:47.589 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13000 IOs, 5692140544 bytes 2026-02-20T22:59:47.664 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13100 IOs, 5767607296 bytes 2026-02-20T22:59:47.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13200 IOs, 5823270912 bytes 2026-02-20T22:59:47.784 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13300 IOs, 5841160192 bytes 2026-02-20T22:59:47.876 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13400 IOs, 5932131328 bytes 2026-02-20T22:59:47.966 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13500 IOs, 5998145536 bytes 2026-02-20T22:59:48.019 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13600 IOs, 6041289728 bytes 2026-02-20T22:59:48.101 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13700 IOs, 6093516800 bytes 2026-02-20T22:59:48.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13800 IOs, 6156353536 bytes 2026-02-20T22:59:48.230 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:13900 IOs, 6242597888 bytes 2026-02-20T22:59:48.306 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14000 IOs, 6306115584 bytes 2026-02-20T22:59:48.369 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14100 IOs, 6344908800 bytes 2026-02-20T22:59:48.427 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14200 IOs, 6413620224 bytes 2026-02-20T22:59:48.478 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14300 IOs, 6430799872 bytes 2026-02-20T22:59:48.528 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14400 IOs, 6456588288 bytes 2026-02-20T22:59:48.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14500 IOs, 6512728064 bytes 2026-02-20T22:59:48.665 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14600 IOs, 6601317376 bytes 2026-02-20T22:59:48.717 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14700 IOs, 6613463040 bytes 2026-02-20T22:59:48.764 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14800 IOs, 6640927744 bytes 2026-02-20T22:59:48.812 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:14900 IOs, 6665843712 bytes 2026-02-20T22:59:48.847 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15000 IOs, 6693440512 bytes 2026-02-20T22:59:48.921 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15100 IOs, 6736951296 bytes 2026-02-20T22:59:48.957 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15200 IOs, 6746873856 bytes 2026-02-20T22:59:49.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15300 IOs, 6789490688 bytes 2026-02-20T22:59:49.094 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15400 IOs, 6874369024 bytes 2026-02-20T22:59:49.175 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15500 IOs, 6965770240 bytes 2026-02-20T22:59:49.236 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15600 IOs, 7006413824 bytes 2026-02-20T22:59:49.275 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15700 IOs, 7019502592 bytes 2026-02-20T22:59:49.349 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15800 IOs, 7090424832 bytes 2026-02-20T22:59:49.614 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:15900 IOs, 7163884544 bytes 2026-02-20T22:59:49.693 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16000 IOs, 7254621184 bytes 2026-02-20T22:59:49.735 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16100 IOs, 7255219200 bytes 2026-02-20T22:59:49.773 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16200 IOs, 7267853312 bytes 2026-02-20T22:59:49.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16300 IOs, 7347016704 bytes 2026-02-20T22:59:49.911 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16400 IOs, 7387082752 bytes 2026-02-20T22:59:49.965 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16500 IOs, 7437652992 bytes 2026-02-20T22:59:50.011 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16600 IOs, 7470999552 bytes 2026-02-20T22:59:50.085 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16700 IOs, 7520951296 bytes 2026-02-20T22:59:50.146 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16800 IOs, 7568788480 bytes 2026-02-20T22:59:50.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:16900 IOs, 7596888064 bytes 2026-02-20T22:59:50.250 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17000 IOs, 7649906688 bytes 2026-02-20T22:59:50.292 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17100 IOs, 7669147648 bytes 2026-02-20T22:59:50.340 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17200 IOs, 7686894592 bytes 2026-02-20T22:59:50.391 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17300 IOs, 7730190336 bytes 2026-02-20T22:59:50.451 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17400 IOs, 7762615296 bytes 2026-02-20T22:59:50.495 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17500 IOs, 7778114560 bytes 2026-02-20T22:59:50.566 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17600 IOs, 7846443008 bytes 2026-02-20T22:59:50.631 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17700 IOs, 7916335104 bytes 2026-02-20T22:59:50.685 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17800 IOs, 7922614272 bytes 2026-02-20T22:59:50.741 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:17900 IOs, 7943083008 bytes 2026-02-20T22:59:50.792 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18000 IOs, 7985061888 bytes 2026-02-20T22:59:50.837 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18100 IOs, 8007404544 bytes 2026-02-20T22:59:50.906 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18200 IOs, 8060160000 bytes 2026-02-20T22:59:50.948 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18300 IOs, 8070942720 bytes 2026-02-20T22:59:51.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:18400 IOs, 8125590528 bytes 2026-02-20T22:59:51.042 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:51.043 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:59:51.044 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12b1ca963315 2026-02-20T22:59:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:59:51.071 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_9 present 12b1ca963315 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12b1ca963315 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:51.172 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:59:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:59:51.173 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:51.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12b1ca963315 2026-02-20T22:59:51.201 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12b1ca963315 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12b1ca963315 -o 12b1ca963315 = 12b1ca963315 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_9 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:51.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2026-02-20T22:59:52.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:59:51.993+0000 7f2b66f8a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619820b5800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2026-02-20T22:59:52.000 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T22:59:51.993+0000 7f2b66f8a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619820b5800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2026-02-20T22:59:56.353 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6760045676990492) > weight (0.25) 2026-02-20T22:59:56.353 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2026-02-20T22:59:56.353 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2026-02-20T22:59:56.353 INFO:tasks.rbd_mirror_thrash:waiting for 5.295091474077884 secs before reviving daemons 2026-02-20T22:59:56.354 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T22:59:56.349+0000 7f93285d8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38215) UID: 1000 2026-02-20T22:59:57.002 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:59:57.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:57.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:57.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:59:57.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:59:57.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:57.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-20T22:59:57.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T22:59:57.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T22:59:57.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_9 2026-02-20T22:59:57.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T22:59:57.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T22:59:57.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_9 2026-02-20T22:59:57.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2026-02-20T22:59:57.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T22:59:57.160 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_9: (2) No such file or directory 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_9 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:57.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_9 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_9 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T22:59:57.198 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_9 2026-02-20T22:59:57.199 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2026-02-20T22:59:57.235 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2026-02-20T23:00:01.649 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2026-02-20T23:00:01.649 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T23:00:01.649 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2026-02-20T23:00:01.649 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2026-02-20T23:00:01.649 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2026-02-20T23:00:01.651 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2026-02-20T23:00:01.651 INFO:tasks.rbd_mirror_thrash:waiting for 93.32199418409328 secs before thrashing 2026-02-20T23:00:21.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T23:00:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for i in `seq 1 10` 2026-02-20T23:00:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image=image_10 2026-02-20T23:00:21.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ create_image cluster2 mirror image_10 512M 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T23:00:21.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local size=128 2026-02-20T23:00:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -n 512M ']' 2026-02-20T23:00:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ size=512M 2026-02-20T23:00:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ shift 2026-02-20T23:00:21.313 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2026-02-20T23:00:21.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ enable_mirror cluster2 mirror image_10 2026-02-20T23:00:21.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local mode=snapshot 2026-02-20T23:00:21.344 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2026-02-20T23:00:22.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring enabled 2026-02-20T23:00:22.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2026-02-20T23:00:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:rbd image 'image_10': 2026-02-20T23:00:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: size 512 MiB in 128 objects 2026-02-20T23:00:22.031 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: order 22 (4 MiB objects) 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: snapshot_count: 1 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: id: 19839717d0c5 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: block_name_prefix: rbd_data.19839717d0c5 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: format: 2 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: features: layering, exclusive-lock 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: op_features: 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: flags: 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: create_timestamp: Fri Feb 20 23:00:21 2026 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: access_timestamp: Fri Feb 20 23:00:21 2026 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: modify_timestamp: Fri Feb 20 23:00:21 2026 2026-02-20T23:00:22.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring state: enabled 2026-02-20T23:00:22.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring mode: snapshot 2026-02-20T23:00:22.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring global id: 6dcbead1-7301-49c9-9c06-541b5e09adb9 2026-02-20T23:00:22.033 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout: mirroring primary: true 2026-02-20T23:00:22.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stress_write_image cluster2 mirror image_10 2026-02-20T23:00:22.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:22.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:22.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:22.034 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2026-02-20T23:00:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local duration=6 2026-02-20T23:00:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:22.036 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2026-02-20T23:00:22.156 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:100 IOs, 95541248 bytes 2026-02-20T23:00:22.215 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:200 IOs, 150415360 bytes 2026-02-20T23:00:22.288 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:300 IOs, 236718080 bytes 2026-02-20T23:00:22.358 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:400 IOs, 279580672 bytes 2026-02-20T23:00:22.414 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:500 IOs, 327173120 bytes 2026-02-20T23:00:22.460 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:600 IOs, 358731776 bytes 2026-02-20T23:00:22.531 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:700 IOs, 417695744 bytes 2026-02-20T23:00:22.570 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:800 IOs, 427523072 bytes 2026-02-20T23:00:22.606 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:900 IOs, 431257600 bytes 2026-02-20T23:00:22.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1000 IOs, 492139520 bytes 2026-02-20T23:00:22.749 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1100 IOs, 565700608 bytes 2026-02-20T23:00:22.817 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1200 IOs, 629254144 bytes 2026-02-20T23:00:22.904 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1300 IOs, 739399680 bytes 2026-02-20T23:00:22.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1400 IOs, 782175232 bytes 2026-02-20T23:00:23.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1500 IOs, 802325504 bytes 2026-02-20T23:00:23.066 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1600 IOs, 815418368 bytes 2026-02-20T23:00:23.140 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1700 IOs, 885760000 bytes 2026-02-20T23:00:23.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1800 IOs, 924002304 bytes 2026-02-20T23:00:23.264 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:1900 IOs, 980770816 bytes 2026-02-20T23:00:23.343 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2000 IOs, 1062881280 bytes 2026-02-20T23:00:23.402 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2100 IOs, 1091781632 bytes 2026-02-20T23:00:23.457 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2200 IOs, 1139435520 bytes 2026-02-20T23:00:23.514 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2300 IOs, 1168453632 bytes 2026-02-20T23:00:23.546 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2400 IOs, 1190477824 bytes 2026-02-20T23:00:23.600 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2500 IOs, 1222356992 bytes 2026-02-20T23:00:23.672 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2600 IOs, 1279033344 bytes 2026-02-20T23:00:23.755 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2700 IOs, 1370216448 bytes 2026-02-20T23:00:23.816 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2800 IOs, 1372017664 bytes 2026-02-20T23:00:23.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:2900 IOs, 1393739776 bytes 2026-02-20T23:00:23.896 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3000 IOs, 1414022144 bytes 2026-02-20T23:00:23.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3100 IOs, 1479358464 bytes 2026-02-20T23:00:24.041 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3200 IOs, 1544790016 bytes 2026-02-20T23:00:24.125 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3300 IOs, 1614256128 bytes 2026-02-20T23:00:24.192 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3400 IOs, 1668507648 bytes 2026-02-20T23:00:24.242 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3500 IOs, 1701112832 bytes 2026-02-20T23:00:24.312 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3600 IOs, 1759133696 bytes 2026-02-20T23:00:24.394 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3700 IOs, 1839521792 bytes 2026-02-20T23:00:24.453 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3800 IOs, 1883984896 bytes 2026-02-20T23:00:24.500 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:3900 IOs, 1924389888 bytes 2026-02-20T23:00:24.543 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4000 IOs, 1928019968 bytes 2026-02-20T23:00:24.596 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4100 IOs, 1970800640 bytes 2026-02-20T23:00:24.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4200 IOs, 2006592512 bytes 2026-02-20T23:00:24.695 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4300 IOs, 2033208320 bytes 2026-02-20T23:00:24.771 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4400 IOs, 2106920960 bytes 2026-02-20T23:00:24.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4500 IOs, 2154641408 bytes 2026-02-20T23:00:24.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4600 IOs, 2155237376 bytes 2026-02-20T23:00:24.934 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4700 IOs, 2184391680 bytes 2026-02-20T23:00:25.015 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4800 IOs, 2253570048 bytes 2026-02-20T23:00:25.068 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:4900 IOs, 2303812608 bytes 2026-02-20T23:00:25.111 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5000 IOs, 2357400576 bytes 2026-02-20T23:00:25.177 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5100 IOs, 2386135040 bytes 2026-02-20T23:00:25.228 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5200 IOs, 2427059200 bytes 2026-02-20T23:00:25.311 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5300 IOs, 2514709504 bytes 2026-02-20T23:00:25.371 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5400 IOs, 2553771008 bytes 2026-02-20T23:00:25.419 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5500 IOs, 2568850432 bytes 2026-02-20T23:00:25.488 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5600 IOs, 2639269888 bytes 2026-02-20T23:00:25.576 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5700 IOs, 2723562496 bytes 2026-02-20T23:00:25.612 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5800 IOs, 2732503040 bytes 2026-02-20T23:00:25.675 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:5900 IOs, 2787115008 bytes 2026-02-20T23:00:25.743 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6000 IOs, 2840687616 bytes 2026-02-20T23:00:25.803 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6100 IOs, 2885955584 bytes 2026-02-20T23:00:25.851 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6200 IOs, 2888898560 bytes 2026-02-20T23:00:25.898 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6300 IOs, 2912124928 bytes 2026-02-20T23:00:25.960 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6400 IOs, 2961446912 bytes 2026-02-20T23:00:26.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6500 IOs, 3024330752 bytes 2026-02-20T23:00:26.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6600 IOs, 3032156160 bytes 2026-02-20T23:00:26.141 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6700 IOs, 3108023296 bytes 2026-02-20T23:00:26.188 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6800 IOs, 3120484352 bytes 2026-02-20T23:00:26.224 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:6900 IOs, 3131996160 bytes 2026-02-20T23:00:26.258 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7000 IOs, 3132719104 bytes 2026-02-20T23:00:26.305 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7100 IOs, 3159288832 bytes 2026-02-20T23:00:26.368 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7200 IOs, 3242809344 bytes 2026-02-20T23:00:26.442 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7300 IOs, 3290121216 bytes 2026-02-20T23:00:26.486 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7400 IOs, 3310313472 bytes 2026-02-20T23:00:26.534 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7500 IOs, 3355190272 bytes 2026-02-20T23:00:26.593 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7600 IOs, 3397919744 bytes 2026-02-20T23:00:26.646 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7700 IOs, 3439353856 bytes 2026-02-20T23:00:26.701 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7800 IOs, 3477480448 bytes 2026-02-20T23:00:26.748 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:7900 IOs, 3504074752 bytes 2026-02-20T23:00:26.824 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8000 IOs, 3582474240 bytes 2026-02-20T23:00:26.887 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8100 IOs, 3622144000 bytes 2026-02-20T23:00:26.943 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8200 IOs, 3668943872 bytes 2026-02-20T23:00:27.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8300 IOs, 3756102656 bytes 2026-02-20T23:00:27.109 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8400 IOs, 3849861120 bytes 2026-02-20T23:00:27.202 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8500 IOs, 3916011520 bytes 2026-02-20T23:00:27.301 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8600 IOs, 3996207104 bytes 2026-02-20T23:00:27.350 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8700 IOs, 4025776128 bytes 2026-02-20T23:00:27.450 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8800 IOs, 4114063360 bytes 2026-02-20T23:00:27.542 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:8900 IOs, 4228974592 bytes 2026-02-20T23:00:27.594 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9000 IOs, 4270685184 bytes 2026-02-20T23:00:27.647 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9100 IOs, 4327477248 bytes 2026-02-20T23:00:27.709 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9200 IOs, 4379666432 bytes 2026-02-20T23:00:27.789 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9300 IOs, 4472147968 bytes 2026-02-20T23:00:27.829 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9400 IOs, 4486575104 bytes 2026-02-20T23:00:27.908 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9500 IOs, 4570721280 bytes 2026-02-20T23:00:27.984 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:9600 IOs, 4662137856 bytes 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=124 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 124 -eq 124 ']' 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2026-02-20T23:00:28.045 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=present 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T23:00:28.046 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T23:00:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id=12b1e96be8d6 2026-02-20T23:00:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T23:00:28.076 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T23:00:28.178 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_10 present 12b1e96be8d6 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=present 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id=12b1e96be8d6 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T23:00:28.179 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T23:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T23:00:28.180 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T23:00:28.209 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id=12b1e96be8d6 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 12b1e96be8d6 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -z 12b1e96be8d6 -o 12b1e96be8d6 = 12b1e96be8d6 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_state=present 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test present = present 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ disable_mirror cluster2 mirror image_10 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:28.210 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2026-02-20T23:00:28.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:00:28.988+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619824a3800 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2026-02-20T23:00:28.996 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:00:28.988+0000 7f2b6778b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x5619824a3800 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=c3ececfc-6db9-428b-b666-3a05930e41b3, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2026-02-20T23:00:34.000 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:Mirroring disabled 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local state=deleted 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T23:00:34.008 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local s 2026-02-20T23:00:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T23:00:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T23:00:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:34.009 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T23:00:34.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T23:00:34.010 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T23:00:34.032 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-20T23:00:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ image_id= 2026-02-20T23:00:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2026-02-20T23:00:34.035 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0.1 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local test_state=deleted 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image_id= 2026-02-20T23:00:34.137 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_state=deleted 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local current_image_id 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ get_image_id cluster1 mirror image_10 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local image=image_10 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2026-02-20T23:00:34.138 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2026-02-20T23:00:34.159 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:rbd: error opening image image_10: (2) No such file or directory 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ current_image_id= 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n '' 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test deleted = deleted 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ purge_snapshots cluster2 mirror image_10 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:34.162 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:34.163 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2026-02-20T23:00:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image_retry cluster2 mirror image_10 2026-02-20T23:00:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:34.194 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 4 8 16 32 2026-02-20T23:00:34.195 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ remove_image cluster2 mirror image_10 2026-02-20T23:00:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:34.196 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local image=image_10 2026-02-20T23:00:34.197 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2026-02-20T23:00:34.226 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2026-02-20T23:00:48.490 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2026-02-20T23:00:48.491 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2026-02-20T23:00:48.491 INFO:tasks.rbd_mirror_thrash:waiting for 1.7312240933027656 secs before reviving daemons 2026-02-20T23:00:48.509 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:00:48.484+0000 7f2b74fa6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 34553) UID: 1000 2026-02-20T23:00:50.222 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2026-02-20T23:00:50.223 INFO:teuthology.orchestra.run:waiting for 600 2026-02-20T23:00:50.223 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2026-02-20T23:00:50.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2026-02-20T23:00:50.223 DEBUG:teuthology.orchestra.run.trial107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2026-02-20T23:00:50.224 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2026-02-20T23:00:50.225 INFO:tasks.rbd_mirror_thrash:waiting for 81.19943777675462 secs before thrashing 2026-02-20T23:00:58.918 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2026-02-20T23:00:58.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:58.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2026-02-20T23:00:58.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ tee -a /tmp/tmp.l7jgOKLHDi/rbd-mirror.test.log 2026-02-20T23:00:58.923 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ date '+%F %T' 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo 2026-02-20 23:00:58 TEST: check if removed 'images'\''' OMAP are removed 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:2026-02-20 23:00:58 TEST: check if removed images' OMAP are removed 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_in_omap cluster1 mirror 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:58.925 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirroring 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=status_global 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:58.926 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=status_global 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c status_global 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:58.927 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:58.953 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirroring 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=image_ 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:58.954 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=image_ 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:58.955 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2026-02-20T23:00:58.956 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c image_ 2026-02-20T23:00:58.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:58.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:58.979 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirror_leader 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=image_map 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:58.980 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirror_leader 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=image_map 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c image_map 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster1 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:58.981 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirror_leader 2026-02-20T23:00:58.982 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2026-02-20T23:00:59.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:59.003 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_image_in_omap cluster2 mirror 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirroring 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=status_global 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:59.004 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=status_global 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.006 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c status_global 2026-02-20T23:00:59.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:59.007 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirroring 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=image_ 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:59.026 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:59.027 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=image_ 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirroring 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c image_ 2026-02-20T23:00:59.028 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:00:59.050 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local pool=mirror 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local obj_name=rbd_mirror_leader 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local filter=image_map 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ sleep 0 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirror_leader 2026-02-20T23:00:59.051 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local filter=image_map 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ grep -c image_map 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local cluster=cluster2 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local pool=mirror 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ local obj_name=rbd_mirror_leader 2026-02-20T23:00:59.052 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2026-02-20T23:00:59.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test 0 = 0 2026-02-20T23:00:59.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ error_code=0 2026-02-20T23:00:59.073 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set -e 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 0 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ cleanup 0 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local error_code=0 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ set +e 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -ne 0 ']' 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z '' ']' 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster instance 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T23:00:59.074 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-20T23:00:59.861 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror' does not exist 2026-02-20T23:00:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T23:00:59.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2026-02-20T23:01:00.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror' does not exist 2026-02-20T23:01:00.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T23:01:00.883 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-20T23:01:01.737 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d7f31640 -1 rbd::mirror::RemotePoolPoller: 0x560bdc70cdd0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2026-02-20T23:01:01.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d8732640 -1 rbd::mirror::RemotePoolPoller: 0x560bdc70cdd0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2026-02-20T23:01:01.738 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d8732640 -1 rbd::mirror::PoolReplayer: 0x560bdd67c000 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2026-02-20T23:01:01.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::InstanceWatcher: 0x560bde20b500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6527 object: (2) No such file or directory 2026-02-20T23:01:01.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::InstanceWatcher: 0x560bdd8c2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6527 object: (2) No such file or directory 2026-02-20T23:01:01.740 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::InstanceWatcher: 0x560bde20b180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6527 object: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30def3f640 -1 rbd::mirror::InstanceWatcher: 0x560bde20b500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30def3f640 -1 rbd::mirror::InstanceWatcher: 0x560bdd8c2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30def3f640 -1 rbd::mirror::InstanceWatcher: 0x560bde20b180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d8732640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd868480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d8732640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd868360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30d8732640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd8686c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86a000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86a1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e574c640 -1 rbd::mirror::NamespaceReplayer: 0x560bdc71ee00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.741 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30de73e640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd868120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30de73e640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd868240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30de73e640 -1 rbd::mirror::MirrorStatusUpdater 0x560bdd868000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:01.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e4f4b640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86a000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e4f4b640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86a1c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.742 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.732+0000 7f30e4f4b640 -1 rbd::mirror::NamespaceReplayer: 0x560bdc71ee00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:01.811 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:01.804+0000 7f30e574c640 -1 rbd::mirror::LeaderWatcher: 0x560bde552300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:01.867 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror_parent' does not exist 2026-02-20T23:01:01.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ CEPH_ARGS= 2026-02-20T23:01:01.877 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2026-02-20T23:01:02.853 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:pool 'mirror_parent' does not exist 2026-02-20T23:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-20T23:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirrors cluster1 2026-02-20T23:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1 2026-02-20T23:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.865 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 0 3 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster1:0 '' 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:0 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster1:1 '' 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:1 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster1:2 '' 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:2 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster1:3 '' 2026-02-20T23:01:02.868 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster1:3 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirrors cluster2 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:++ seq 0 3 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster2:0 '' 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:0 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.869 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster2:1 '' 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:1 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster2:2 '' 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:2 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ stop_mirror cluster2:3 '' 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local cluster=cluster2:3 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ local sig= 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test -n 1 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ return 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' -z 1 ']' 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ test '' = /tmp/tmp.l7jgOKLHDi 2026-02-20T23:01:02.870 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ rm -Rf /tmp/tmp.l7jgOKLHDi 2026-02-20T23:01:02.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ '[' 0 -eq 0 ']' 2026-02-20T23:01:02.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ echo OK 2026-02-20T23:01:02.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stdout:OK 2026-02-20T23:01:02.872 INFO:tasks.workunit.cluster1.client.mirror.trial107.stderr:+ exit 0 2026-02-20T23:01:02.872 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-20T23:01:02.873 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2026-02-20T23:01:02.922 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2026-02-20T23:01:02.922 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2026-02-20T23:01:03.274 DEBUG:teuthology.parallel:result is None 2026-02-20T23:01:03.274 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-20T23:01:03.283 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-20T23:01:03.283 DEBUG:teuthology.orchestra.run.trial107:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2026-02-20T23:01:03.330 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2026-02-20T23:01:03.330 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2026-02-20T23:01:03.349 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2026-02-20T23:01:03.349 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 17 2026-02-20T23:01:03.349 INFO:tasks.rbd_mirror_thrash:done joining 2026-02-20T23:01:03.349 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2026-02-20T23:01:03.366 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2026-02-20T23:01:03.366 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 14 2026-02-20T23:01:03.366 INFO:tasks.rbd_mirror_thrash:done joining 2026-02-20T23:01:03.367 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.390 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2026-02-20T23:01:03.390 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.394 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9d63640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 31467) UID: 1000 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558910ce58c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558910964000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x558910ce5d40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::PoolWatcher: 0x558910e40780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::PoolWatcher: 0x558910e403c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::PoolWatcher: 0x558910e405a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::InstanceWatcher: 0x558910caa380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5017 object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::InstanceWatcher: 0x5589107fbc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5017 object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edbd47640 -1 rbd::mirror::PoolWatcher: 0x558910f3c000 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edbd47640 -1 rbd::mirror::PoolWatcher: 0x558910f3d0e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edbd47640 -1 rbd::mirror::PoolWatcher: 0x558910e40960 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.395 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ed4d39640 -1 rbd::mirror::InstanceWatcher: 0x558910caa380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ed4d39640 -1 rbd::mirror::InstanceWatcher: 0x5589107fbc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::InstanceWatcher: 0x5589107fb880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5014 object: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::InstanceWatcher: 0x5589111a1180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5014 object: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::InstanceWatcher: 0x5589111a1500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5014 object: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8eced2d640 -1 rbd::mirror::MirrorStatusUpdater 0x558910a72ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8eced2d640 -1 rbd::mirror::MirrorStatusUpdater 0x558910a72d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081d180 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081d340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ed4d39640 -1 rbd::mirror::MirrorStatusUpdater 0x558910a72a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ed4d39640 -1 rbd::mirror::MirrorStatusUpdater 0x558910a72b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::NamespaceReplayer: 0x55891081d180 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081d340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::InstanceWatcher: 0x5589111a1180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::InstanceWatcher: 0x5589111a1500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.396 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::InstanceWatcher: 0x5589107fb880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::LeaderWatcher: 0x558910addb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edc548640 -1 rbd::mirror::MirrorStatusUpdater 0x558910818360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edc548640 -1 rbd::mirror::MirrorStatusUpdater 0x5589108185a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081c000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8edc548640 -1 rbd::mirror::MirrorStatusUpdater 0x5589108186c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55890f6b2700 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081c1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2d55640 -1 rbd::mirror::MirrorStatusUpdater 0x558910818120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::NamespaceReplayer: 0x55891081c000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::MirrorStatusUpdater 0x558910818000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee2554640 -1 rbd::mirror::MirrorStatusUpdater 0x558910818240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55890f6b2700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.397 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::NamespaceReplayer: 0x55891081c1c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::LeaderWatcher: 0x55891136a300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-20T23:01:03.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee9562640 -1 rbd::mirror::LeaderWatcher: 0x55891136a300 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2026-02-20T23:01:03.398 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.388+0000 7f8ee8d61640 -1 rbd::mirror::LeaderWatcher: 0x55891136a300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.409 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2026-02-20T23:01:03.409 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.424 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2026-02-20T23:01:03.424 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.450 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff449ca0640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 54169) UID: 1000 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff435477640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557472d5d440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff435477640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x557472d5d200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff434c76640 -1 rbd::mirror::PoolWatcher: 0x557472a4ef00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff435477640 -1 rbd::mirror::PoolWatcher: 0x557472acc1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff442c92640 -1 rbd::mirror::InstanceWatcher: 0x557472416000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff442c92640 -1 rbd::mirror::InstanceWatcher: 0x557472416380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff442c92640 -1 rbd::mirror::InstanceWatcher: 0x557472de7180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.452 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff44949f640 -1 rbd::mirror::InstanceWatcher: 0x55747288e380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6459 object: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff44949f640 -1 rbd::mirror::InstanceWatcher: 0x557472de6380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6459 object: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff434c76640 -1 rbd::mirror::InstanceWatcher: 0x55747288e380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff43c485640 -1 rbd::mirror::MirrorStatusUpdater 0x55747240c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff434c76640 -1 rbd::mirror::InstanceWatcher: 0x557472de6380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff43c485640 -1 rbd::mirror::MirrorStatusUpdater 0x55747240c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff43c485640 -1 rbd::mirror::MirrorStatusUpdater 0x55747240c480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff448c9e640 -1 rbd::mirror::NamespaceReplayer: 0x5574712f2e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff448c9e640 -1 rbd::mirror::NamespaceReplayer: 0x55747240e1c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff448c9e640 -1 rbd::mirror::NamespaceReplayer: 0x55747240e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff435477640 -1 rbd::mirror::MirrorStatusUpdater 0x5574726be900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff44949f640 -1 rbd::mirror::NamespaceReplayer: 0x55747240efc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff434c76640 -1 rbd::mirror::MirrorStatusUpdater 0x5574726bea20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff448c9e640 -1 rbd::mirror::NamespaceReplayer: 0x55747240f180 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.453 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff44949f640 -1 rbd::mirror::LeaderWatcher: 0x5574726ebb00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-20T23:01:03.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff44949f640 -1 rbd::mirror::LeaderWatcher: 0x5574726ebb00 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2026-02-20T23:01:03.454 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.444+0000 7ff448c9e640 -1 rbd::mirror::LeaderWatcher: 0x5574726ebb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.467 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2026-02-20T23:01:03.468 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.488 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2026-02-20T23:01:03.488 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.514 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e5f4d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 55106) UID: 1000 2026-02-20T23:01:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e4f4b640 -1 rbd::mirror::InstanceWatcher: 0x560bddd06380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6529 object: (2) No such file or directory 2026-02-20T23:01:03.515 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e4f4b640 -1 rbd::mirror::InstanceWatcher: 0x560bdd8c2380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6529 object: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30d1724640 -1 rbd::mirror::InstanceWatcher: 0x560bddd06380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30d1724640 -1 rbd::mirror::InstanceWatcher: 0x560bdd8c2380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30d1724640 -1 rbd::mirror::MirrorStatusUpdater 0x560bddb10900 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30d1724640 -1 rbd::mirror::MirrorStatusUpdater 0x560bddb10a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e4f4b640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86afc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e4f4b640 -1 rbd::mirror::NamespaceReplayer: 0x560bdd86b180 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.516 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f30e574c640 -1 rbd::mirror::LeaderWatcher: 0x560bddb3db00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f202b358640 -1 rbd::mirror::LeaderWatcher: 0x565417df9b00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2026-02-20T23:01:03.517 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2016b2f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x565417aac000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2016b2f640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x5654187fd200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2017330640 -1 rbd::mirror::PoolWatcher: 0x5654180b54a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2016b2f640 -1 rbd::mirror::PoolWatcher: 0x5654180b5680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2010322640 -1 rbd::mirror::PoolWatcher: 0x5654180b5c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f2010b23640 -1 rbd::mirror::PoolWatcher: 0x5654180b5860 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f202ab57640 -1 rbd::mirror::LeaderWatcher: 0x565417df9b00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-20T23:01:03.518 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.508+0000 7f202ab57640 -1 rbd::mirror::LeaderWatcher: 0x565417df9b00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2026-02-20T23:01:03.524 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe5ca85640 -1 rbd::mirror::LeaderWatcher: 0x560b27de2300 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2026-02-20T23:01:03.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56a79640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560b275c38c0 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56278640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560b27d8b200 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.525 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56278640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560b27d8b440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2026-02-20T23:01:03.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56a79640 -1 rbd::mirror::PoolWatcher: 0x560b25f73c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56278640 -1 rbd::mirror::PoolWatcher: 0x560b27d732c0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe56a79640 -1 rbd::mirror::PoolWatcher: 0x560b27d721e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2026-02-20T23:01:03.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe5d286640 -1 rbd::mirror::LeaderWatcher: 0x560b27de2300 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2026-02-20T23:01:03.526 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.516+0000 7fbe5d286640 -1 rbd::mirror::LeaderWatcher: 0x560b27de2300 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2026-02-20T23:01:03.528 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2026-02-20T23:01:03.528 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.546 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2026-02-20T23:01:03.546 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.570 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d6d00640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 53372) UID: 1000 2026-02-20T23:01:03.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82cfcf2640 -1 rbd::mirror::InstanceWatcher: 0x559e6b919c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.573 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82cfcf2640 -1 rbd::mirror::InstanceWatcher: 0x559e6b918700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82cfcf2640 -1 rbd::mirror::InstanceWatcher: 0x559e6b550380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::InstanceWatcher: 0x559e6b007c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6390 object: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::InstanceWatcher: 0x559e6b550000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6390 object: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c1cd6640 -1 rbd::mirror::InstanceWatcher: 0x559e6b007c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c1cd6640 -1 rbd::mirror::InstanceWatcher: 0x559e6b550000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c8ce4640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6af465a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c8ce4640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6af46480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c8ce4640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6af46360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e69e24e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af481c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af48000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.574 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82cf4f1640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6af46000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82cf4f1640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6af46120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e69e24e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af48000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82bb4c9640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6b1c0d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af49180 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82bbcca640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6adeb320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d5cfe640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af49340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c1cd6640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6b1c0b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82c1cd6640 -1 rbd::mirror::MirrorStatusUpdater 0x559e6b1c0a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af49340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d64ff640 -1 rbd::mirror::NamespaceReplayer: 0x559e6af49180 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.575 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.564+0000 7f82d5cfe640 -1 rbd::mirror::LeaderWatcher: 0x559e6b21fb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.576 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.568+0000 7f82d5cfe640 -1 rbd::mirror::LeaderWatcher: 0x559e6bb20300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.587 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2026-02-20T23:01:03.587 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.608 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2026-02-20T23:01:03.609 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005aa9c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 54763) UID: 1000 2026-02-20T23:01:03.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::InstanceWatcher: 0x55a6a0274380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6018 object: (2) No such file or directory 2026-02-20T23:01:03.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdffc00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6018 object: (2) No such file or directory 2026-02-20T23:01:03.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdfe000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6016 object: (2) No such file or directory 2026-02-20T23:01:03.634 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdff880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6016 object: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::InstanceWatcher: 0x55a6a0763180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6016 object: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0046273640 -1 rbd::mirror::InstanceWatcher: 0x55a6a0274380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0045a72640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdffc00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005328d640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdfe000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005328d640 -1 rbd::mirror::InstanceWatcher: 0x55a69fdff880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f003f265640 -1 rbd::mirror::MirrorStatusUpdater 0x55a6a00190e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f003f265640 -1 rbd::mirror::MirrorStatusUpdater 0x55a6a0018d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ff3e700 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0053a8e640 -1 rbd::mirror::InstanceWatcher: 0x55a6a0763180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ff3e8c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.635 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0046273640 -1 rbd::mirror::MirrorStatusUpdater 0x55a6a0018a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ff3e700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f004d281640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f004d281640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f004d281640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82fc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0045a72640 -1 rbd::mirror::MirrorStatusUpdater 0x55a6a0018b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82700 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ff3e8c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005328d640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82fc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::LeaderWatcher: 0x55a6a00a7b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0053a8e640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f0059a9a640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005328d640 -1 rbd::mirror::MirrorStatusUpdater 0x55a69fdba120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.636 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.628+0000 7f005a29b640 -1 rbd::mirror::NamespaceReplayer: 0x55a69ec82000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.637 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.trial107.stderr:2026-02-20T23:01:03.632+0000 7f0059a9a640 -1 rbd::mirror::LeaderWatcher: 0x55a6a0aac300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.650 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2026-02-20T23:01:03.650 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.663 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2026-02-20T23:01:03.663 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.694 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5da87640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 53972) UID: 1000 2026-02-20T23:01:03.696 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::InstanceWatcher: 0x560b27176000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5971 object: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::InstanceWatcher: 0x560b27ab7180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5971 object: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::InstanceWatcher: 0x560b27177500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5974 object: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::InstanceWatcher: 0x560b27177c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5974 object: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::InstanceWatcher: 0x560b27176700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5971 object: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56a79640 -1 rbd::mirror::InstanceWatcher: 0x560b27176000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56278640 -1 rbd::mirror::InstanceWatcher: 0x560b27ab7180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe48a5d640 -1 rbd::mirror::InstanceWatcher: 0x560b27177500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56a79640 -1 rbd::mirror::InstanceWatcher: 0x560b27176700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe4925e640 -1 rbd::mirror::InstanceWatcher: 0x560b27177c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5026c640 -1 rbd::mirror::MirrorStatusUpdater 0x560b2710c5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5026c640 -1 rbd::mirror::MirrorStatusUpdater 0x560b2710c480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::NamespaceReplayer: 0x560b25fbc700 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::NamespaceReplayer: 0x560b25fbca80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.697 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe42250640 -1 rbd::mirror::MirrorStatusUpdater 0x560b273750e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe42250640 -1 rbd::mirror::MirrorStatusUpdater 0x560b27374d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56278640 -1 rbd::mirror::MirrorStatusUpdater 0x560b2710c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::NamespaceReplayer: 0x560b27e9a8c0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::NamespaceReplayer: 0x560b27e9aa80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56a79640 -1 rbd::mirror::MirrorStatusUpdater 0x560b2710c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::NamespaceReplayer: 0x560b25fbce00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::NamespaceReplayer: 0x560b25fbc700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe56278640 -1 rbd::mirror::MirrorStatusUpdater 0x560b2710c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5d286640 -1 rbd::mirror::NamespaceReplayer: 0x560b25fbca80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe48a5d640 -1 rbd::mirror::MirrorStatusUpdater 0x560b27374a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::NamespaceReplayer: 0x560b27e9a8c0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe4925e640 -1 rbd::mirror::MirrorStatusUpdater 0x560b27374b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::NamespaceReplayer: 0x560b27e9aa80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.698 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.688+0000 7fbe5ca85640 -1 rbd::mirror::LeaderWatcher: 0x560b27405b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.699 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.trial107.stderr:2026-02-20T23:01:03.692+0000 7fbe5ca85640 -1 rbd::mirror::LeaderWatcher: 0x560b27de2300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.711 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2026-02-20T23:01:03.711 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.731 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2026-02-20T23:01:03.731 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.758 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202bb59640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 52524) UID: 1000 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::InstanceWatcher: 0x565417c76700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5894 object: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::InstanceWatcher: 0x565417fce380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5894 object: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::InstanceWatcher: 0x5654184c1180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5891 object: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::InstanceWatcher: 0x565417c76380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5891 object: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::InstanceWatcher: 0x565417c76000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5891 object: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2017330640 -1 rbd::mirror::InstanceWatcher: 0x565417c76700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2016b2f640 -1 rbd::mirror::InstanceWatcher: 0x565417fce380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202434a640 -1 rbd::mirror::InstanceWatcher: 0x5654184c1180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2024b4b640 -1 rbd::mirror::InstanceWatcher: 0x565417c76380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.761 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2010322640 -1 rbd::mirror::MirrorStatusUpdater 0x565417d80ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2010322640 -1 rbd::mirror::MirrorStatusUpdater 0x565417d80d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::NamespaceReplayer: 0x565417c74a80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::NamespaceReplayer: 0x565417c74c40 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202434a640 -1 rbd::mirror::InstanceWatcher: 0x565417c76000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2024b4b640 -1 rbd::mirror::MirrorStatusUpdater 0x565417b1c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::NamespaceReplayer: 0x5654169c6700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2017330640 -1 rbd::mirror::MirrorStatusUpdater 0x565417d80a20 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::NamespaceReplayer: 0x565417c74a80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2016b2f640 -1 rbd::mirror::MirrorStatusUpdater 0x565417d80b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202b358640 -1 rbd::mirror::NamespaceReplayer: 0x565417c74c40 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202434a640 -1 rbd::mirror::MirrorStatusUpdater 0x565417b1c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.762 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::NamespaceReplayer: 0x5654169c6e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f2024b4b640 -1 rbd::mirror::MirrorStatusUpdater 0x565417b1c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::NamespaceReplayer: 0x5654169c6a80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::LeaderWatcher: 0x565417df9b00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.763 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.trial107.stderr:2026-02-20T23:01:03.752+0000 7f202ab57640 -1 rbd::mirror::LeaderWatcher: 0x5654187fa300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.775 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2026-02-20T23:01:03.775 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2026-02-20T23:01:03.794 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2026-02-20T23:01:03.794 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:03.818 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e78b1640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 55458) UID: 1000 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::InstanceWatcher: 0x5609809e5500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6053 object: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::InstanceWatcher: 0x5609809e5180 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6053 object: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::InstanceWatcher: 0x5609809e4380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6053 object: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::InstanceWatcher: 0x560980117500 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6056 object: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::InstanceWatcher: 0x560980117880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6056 object: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e08a3640 -1 rbd::mirror::InstanceWatcher: 0x5609809e5500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e00a2640 -1 rbd::mirror::InstanceWatcher: 0x5609809e5180 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e00a2640 -1 rbd::mirror::InstanceWatcher: 0x5609809e4380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.821 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61d2887640 -1 rbd::mirror::InstanceWatcher: 0x560980117500 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61d2887640 -1 rbd::mirror::InstanceWatcher: 0x560980117880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61da096640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b2c480 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61da096640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b2c240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61da096640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b2c360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefd180 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefca80 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefc700 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61cc87b640 -1 rbd::mirror::MirrorStatusUpdater 0x5609802d8360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61cc87b640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b637a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::NamespaceReplayer: 0x560980426e00 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::NamespaceReplayer: 0x560980426fc0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e08a3640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b72000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.822 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefd180 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61d3088640 -1 rbd::mirror::MirrorStatusUpdater 0x5609802d8d80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e00a2640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b2c120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e00a2640 -1 rbd::mirror::MirrorStatusUpdater 0x560980b2c000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x560980426e00 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefca80 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e68af640 -1 rbd::mirror::NamespaceReplayer: 0x56097eefc700 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61d2887640 -1 rbd::mirror::MirrorStatusUpdater 0x5609802d8ea0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2026-02-20T23:01:03.823 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::NamespaceReplayer: 0x560980426fc0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2026-02-20T23:01:03.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.812+0000 7f61e70b0640 -1 rbd::mirror::LeaderWatcher: 0x56098033fb00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.824 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.trial107.stderr:2026-02-20T23:01:03.816+0000 7f61e70b0640 -1 rbd::mirror::LeaderWatcher: 0x560980bfe300 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2026-02-20T23:01:03.837 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2026-02-20T23:01:03.837 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-20T23:01:03.855 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2026-02-20T23:01:03.856 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T23:01:04.027 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:04.027 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T23:01:04.037 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":1528,"stamp":"2026-02-20T23:01:03.741989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4275280,"kb_used_data":16928,"kb_used_omap":1397,"kb_used_meta":4256906,"kb_avail":278840240,"statfs":{"total":289910292480,"available":285532405760,"internally_reserved":0,"allocated":17334272,"data_stored":9097997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1431547,"internal_metadata":4359071749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536621110,"num_objects":-173,"num_object_clones":0,"num_object_copies":-346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-173,"num_whiteouts":0,"num_read":-119090,"num_read_kb":-39715444,"num_write":-1286248,"num_write_kb":-486644537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-16,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806029+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T23:01:00.806029+0000","last_peered":"2026-02-20T23:01:00.806029+0000","last_clean":"2026-02-20T23:01:00.806029+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T23:01:00.806029+0000","last_undegraded":"2026-02-20T23:01:00.806029+0000","last_fullsized":"2026-02-20T23:01:00.806029+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806924+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T23:01:00.806924+0000","last_peered":"2026-02-20T23:01:00.806924+0000","last_clean":"2026-02-20T23:01:00.806924+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T23:01:00.806924+0000","last_undegraded":"2026-02-20T23:01:00.806924+0000","last_fullsized":"2026-02-20T23:01:00.806924+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.809121+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T23:01:00.809121+0000","last_peered":"2026-02-20T23:01:00.809121+0000","last_clean":"2026-02-20T23:01:00.809121+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T23:01:00.809121+0000","last_undegraded":"2026-02-20T23:01:00.809121+0000","last_fullsized":"2026-02-20T23:01:00.809121+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.800318+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T23:01:00.800318+0000","last_peered":"2026-02-20T23:01:00.800318+0000","last_clean":"2026-02-20T23:01:00.800318+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T23:01:00.800318+0000","last_undegraded":"2026-02-20T23:01:00.800318+0000","last_fullsized":"2026-02-20T23:01:00.800318+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.797720+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T23:00:58.797720+0000","last_peered":"2026-02-20T23:00:58.797720+0000","last_clean":"2026-02-20T23:00:58.797720+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T23:00:58.797720+0000","last_undegraded":"2026-02-20T23:00:58.797720+0000","last_fullsized":"2026-02-20T23:00:58.797720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.796737+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T23:00:58.796737+0000","last_peered":"2026-02-20T23:00:58.796737+0000","last_clean":"2026-02-20T23:00:58.796737+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T23:00:58.796737+0000","last_undegraded":"2026-02-20T23:00:58.796737+0000","last_fullsized":"2026-02-20T23:00:58.796737+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.795248+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T23:00:58.795248+0000","last_peered":"2026-02-20T23:00:58.795248+0000","last_clean":"2026-02-20T23:00:58.795248+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T23:00:58.795248+0000","last_undegraded":"2026-02-20T23:00:58.795248+0000","last_fullsized":"2026-02-20T23:00:58.795248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":920,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.807135+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T23:01:00.807135+0000","last_peered":"2026-02-20T23:01:00.807135+0000","last_clean":"2026-02-20T23:01:00.807135+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T23:01:00.807135+0000","last_undegraded":"2026-02-20T23:01:00.807135+0000","last_fullsized":"2026-02-20T23:01:00.807135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1008,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.798989+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T23:01:00.798989+0000","last_peered":"2026-02-20T23:01:00.798989+0000","last_clean":"2026-02-20T23:01:00.798989+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T23:01:00.798989+0000","last_undegraded":"2026-02-20T23:01:00.798989+0000","last_fullsized":"2026-02-20T23:01:00.798989+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1185116,"kb_used_data":5836,"kb_used_omap":122,"kb_used_meta":1179141,"kb_avail":93186724,"statfs":{"total":96636764160,"available":95423205376,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125580,"internal_metadata":1207440756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1271640,"kb_used_data":5256,"kb_used_omap":1070,"kb_used_meta":1265297,"kb_avail":93100200,"statfs":{"total":96636764160,"available":95334604800,"internally_reserved":0,"allocated":5382144,"data_stored":2639087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1096099,"internal_metadata":1295664733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738800,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818524,"kb_used_data":5836,"kb_used_omap":204,"kb_used_meta":1812467,"kb_avail":92553316,"statfs":{"total":96636764160,"available":94774595584,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":209868,"internal_metadata":1855966260},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:04.039 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T23:01:04.195 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:04.196 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T23:01:04.205 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":1528,"stamp":"2026-02-20T23:01:03.741989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4275280,"kb_used_data":16928,"kb_used_omap":1397,"kb_used_meta":4256906,"kb_avail":278840240,"statfs":{"total":289910292480,"available":285532405760,"internally_reserved":0,"allocated":17334272,"data_stored":9097997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1431547,"internal_metadata":4359071749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536621110,"num_objects":-173,"num_object_clones":0,"num_object_copies":-346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-173,"num_whiteouts":0,"num_read":-119090,"num_read_kb":-39715444,"num_write":-1286248,"num_write_kb":-486644537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-16,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806029+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T23:01:00.806029+0000","last_peered":"2026-02-20T23:01:00.806029+0000","last_clean":"2026-02-20T23:01:00.806029+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T23:01:00.806029+0000","last_undegraded":"2026-02-20T23:01:00.806029+0000","last_fullsized":"2026-02-20T23:01:00.806029+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806924+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T23:01:00.806924+0000","last_peered":"2026-02-20T23:01:00.806924+0000","last_clean":"2026-02-20T23:01:00.806924+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T23:01:00.806924+0000","last_undegraded":"2026-02-20T23:01:00.806924+0000","last_fullsized":"2026-02-20T23:01:00.806924+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.809121+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T23:01:00.809121+0000","last_peered":"2026-02-20T23:01:00.809121+0000","last_clean":"2026-02-20T23:01:00.809121+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T23:01:00.809121+0000","last_undegraded":"2026-02-20T23:01:00.809121+0000","last_fullsized":"2026-02-20T23:01:00.809121+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.800318+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T23:01:00.800318+0000","last_peered":"2026-02-20T23:01:00.800318+0000","last_clean":"2026-02-20T23:01:00.800318+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T23:01:00.800318+0000","last_undegraded":"2026-02-20T23:01:00.800318+0000","last_fullsized":"2026-02-20T23:01:00.800318+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.797720+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T23:00:58.797720+0000","last_peered":"2026-02-20T23:00:58.797720+0000","last_clean":"2026-02-20T23:00:58.797720+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T23:00:58.797720+0000","last_undegraded":"2026-02-20T23:00:58.797720+0000","last_fullsized":"2026-02-20T23:00:58.797720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.796737+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T23:00:58.796737+0000","last_peered":"2026-02-20T23:00:58.796737+0000","last_clean":"2026-02-20T23:00:58.796737+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T23:00:58.796737+0000","last_undegraded":"2026-02-20T23:00:58.796737+0000","last_fullsized":"2026-02-20T23:00:58.796737+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.795248+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T23:00:58.795248+0000","last_peered":"2026-02-20T23:00:58.795248+0000","last_clean":"2026-02-20T23:00:58.795248+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T23:00:58.795248+0000","last_undegraded":"2026-02-20T23:00:58.795248+0000","last_fullsized":"2026-02-20T23:00:58.795248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":920,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.807135+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T23:01:00.807135+0000","last_peered":"2026-02-20T23:01:00.807135+0000","last_clean":"2026-02-20T23:01:00.807135+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T23:01:00.807135+0000","last_undegraded":"2026-02-20T23:01:00.807135+0000","last_fullsized":"2026-02-20T23:01:00.807135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1008,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.798989+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T23:01:00.798989+0000","last_peered":"2026-02-20T23:01:00.798989+0000","last_clean":"2026-02-20T23:01:00.798989+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T23:01:00.798989+0000","last_undegraded":"2026-02-20T23:01:00.798989+0000","last_fullsized":"2026-02-20T23:01:00.798989+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1185116,"kb_used_data":5836,"kb_used_omap":122,"kb_used_meta":1179141,"kb_avail":93186724,"statfs":{"total":96636764160,"available":95423205376,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125580,"internal_metadata":1207440756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1271640,"kb_used_data":5256,"kb_used_omap":1070,"kb_used_meta":1265297,"kb_avail":93100200,"statfs":{"total":96636764160,"available":95334604800,"internally_reserved":0,"allocated":5382144,"data_stored":2639087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1096099,"internal_metadata":1295664733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738800,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818524,"kb_used_data":5836,"kb_used_omap":204,"kb_used_meta":1812467,"kb_avail":92553316,"statfs":{"total":96636764160,"available":94774595584,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":209868,"internal_metadata":1855966260},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:04.206 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2026-02-20T23:01:04.206 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T23:01:04.353 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:04.353 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T23:01:04.363 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":1528,"stamp":"2026-02-20T23:01:03.741989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":274,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":4275280,"kb_used_data":16928,"kb_used_omap":1397,"kb_used_meta":4256906,"kb_avail":278840240,"statfs":{"total":289910292480,"available":285532405760,"internally_reserved":0,"allocated":17334272,"data_stored":9097997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1431547,"internal_metadata":4359071749},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536621110,"num_objects":-173,"num_object_clones":0,"num_object_copies":-346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-173,"num_whiteouts":0,"num_read":-119090,"num_read_kb":-39715444,"num_write":-1286248,"num_write_kb":-486644537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-16,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000812"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806029+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T23:01:00.806029+0000","last_peered":"2026-02-20T23:01:00.806029+0000","last_clean":"2026-02-20T23:01:00.806029+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T23:01:00.806029+0000","last_undegraded":"2026-02-20T23:01:00.806029+0000","last_fullsized":"2026-02-20T23:01:00.806029+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806924+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T23:01:00.806924+0000","last_peered":"2026-02-20T23:01:00.806924+0000","last_clean":"2026-02-20T23:01:00.806924+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T23:01:00.806924+0000","last_undegraded":"2026-02-20T23:01:00.806924+0000","last_fullsized":"2026-02-20T23:01:00.806924+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.809121+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T23:01:00.809121+0000","last_peered":"2026-02-20T23:01:00.809121+0000","last_clean":"2026-02-20T23:01:00.809121+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T23:01:00.809121+0000","last_undegraded":"2026-02-20T23:01:00.809121+0000","last_fullsized":"2026-02-20T23:01:00.809121+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.800318+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T23:01:00.800318+0000","last_peered":"2026-02-20T23:01:00.800318+0000","last_clean":"2026-02-20T23:01:00.800318+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T23:01:00.800318+0000","last_undegraded":"2026-02-20T23:01:00.800318+0000","last_fullsized":"2026-02-20T23:01:00.800318+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.797720+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T23:00:58.797720+0000","last_peered":"2026-02-20T23:00:58.797720+0000","last_clean":"2026-02-20T23:00:58.797720+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T23:00:58.797720+0000","last_undegraded":"2026-02-20T23:00:58.797720+0000","last_fullsized":"2026-02-20T23:00:58.797720+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.796737+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T23:00:58.796737+0000","last_peered":"2026-02-20T23:00:58.796737+0000","last_clean":"2026-02-20T23:00:58.796737+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T23:00:58.796737+0000","last_undegraded":"2026-02-20T23:00:58.796737+0000","last_fullsized":"2026-02-20T23:00:58.796737+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":917,"reported_epoch":454,"state":"active+clean","last_fresh":"2026-02-20T23:00:58.795248+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T23:00:58.795248+0000","last_peered":"2026-02-20T23:00:58.795248+0000","last_clean":"2026-02-20T23:00:58.795248+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T23:00:58.795248+0000","last_undegraded":"2026-02-20T23:00:58.795248+0000","last_fullsized":"2026-02-20T23:00:58.795248+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":920,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.807135+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T23:01:00.807135+0000","last_peered":"2026-02-20T23:01:00.807135+0000","last_clean":"2026-02-20T23:01:00.807135+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T23:01:00.807135+0000","last_undegraded":"2026-02-20T23:01:00.807135+0000","last_fullsized":"2026-02-20T23:01:00.807135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1008,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.798989+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T23:01:00.798989+0000","last_peered":"2026-02-20T23:01:00.798989+0000","last_clean":"2026-02-20T23:01:00.798989+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T23:01:00.798989+0000","last_undegraded":"2026-02-20T23:01:00.798989+0000","last_fullsized":"2026-02-20T23:01:00.798989+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1185116,"kb_used_data":5836,"kb_used_omap":122,"kb_used_meta":1179141,"kb_avail":93186724,"statfs":{"total":96636764160,"available":95423205376,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125580,"internal_metadata":1207440756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738801,"num_pgs":81,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1271640,"kb_used_data":5256,"kb_used_omap":1070,"kb_used_meta":1265297,"kb_avail":93100200,"statfs":{"total":96636764160,"available":95334604800,"internally_reserved":0,"allocated":5382144,"data_stored":2639087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1096099,"internal_metadata":1295664733},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738800,"num_pgs":94,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1818524,"kb_used_data":5836,"kb_used_omap":204,"kb_used_meta":1812467,"kb_avail":92553316,"statfs":{"total":96636764160,"available":94774595584,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":209868,"internal_metadata":1855966260},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:04.364 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2026-02-20T23:01:04.521 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:04.521 INFO:teuthology.orchestra.run.trial107.stdout:{"epoch":456,"fsid":"22bff7d5-8e70-4f68-8060-d4046dcd49e4","created":"2026-02-20T22:25:09.821788+0000","modified":"2026-02-20T23:01:02.801338+0000","last_up_change":"2026-02-20T22:25:14.831864+0000","last_in_change":"2026-02-20T22:25:10.448417+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:25:15.471458+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:25:17.609108+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"67b9c177-c488-4711-9869-3c6812b373b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6800","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6801","nonce":1374819874}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6802","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6803","nonce":1374819874}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6806","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6807","nonce":1374819874}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6804","nonce":1374819874},{"type":"v1","addr":"10.20.193.107:6805","nonce":1374819874}]},"public_addr":"10.20.193.107:6801/1374819874","cluster_addr":"10.20.193.107:6803/1374819874","heartbeat_back_addr":"10.20.193.107:6807/1374819874","heartbeat_front_addr":"10.20.193.107:6805/1374819874","state":["exists","up"]},{"osd":1,"uuid":"4ca2fdff-829f-4705-91d4-bf1aa309394c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6808","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6809","nonce":1913959616}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6810","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6811","nonce":1913959616}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6814","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6815","nonce":1913959616}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6812","nonce":1913959616},{"type":"v1","addr":"10.20.193.107:6813","nonce":1913959616}]},"public_addr":"10.20.193.107:6809/1913959616","cluster_addr":"10.20.193.107:6811/1913959616","heartbeat_back_addr":"10.20.193.107:6815/1913959616","heartbeat_front_addr":"10.20.193.107:6813/1913959616","state":["exists","up"]},{"osd":2,"uuid":"b2eeafe2-8f74-402f-9490-aa4cb0904997","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6816","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6817","nonce":2773181681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6818","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6819","nonce":2773181681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6822","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6823","nonce":2773181681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.107:6820","nonce":2773181681},{"type":"v1","addr":"10.20.193.107:6821","nonce":2773181681}]},"public_addr":"10.20.193.107:6817/2773181681","cluster_addr":"10.20.193.107:6819/2773181681","heartbeat_back_addr":"10.20.193.107:6823/2773181681","heartbeat_front_addr":"10.20.193.107:6821/2773181681","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:25:13.130051+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:25:13.219404+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:25:13.298017+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"10.20.193.107:0/1270897782":"2026-02-21T00:00:20.960473+0000","10.20.193.107:0/3538437895":"2026-02-20T23:59:38.872565+0000","10.20.193.107:0/237680807":"2026-02-20T23:58:21.025868+0000","10.20.193.107:0/692529929":"2026-02-20T23:56:59.640608+0000","10.20.193.107:0/2976572352":"2026-02-20T23:55:32.795034+0000","10.20.193.107:0/1994856972":"2026-02-20T23:29:40.268418+0000","10.20.193.107:0/2199019240":"2026-02-20T23:48:12.798520+0000","10.20.193.107:0/675325052":"2026-02-20T23:47:51.742890+0000","10.20.193.107:0/4246005691":"2026-02-20T23:50:19.826341+0000","10.20.193.107:0/3437795565":"2026-02-20T23:47:05.458848+0000","10.20.193.107:0/4235668514":"2026-02-20T23:46:41.412184+0000","10.20.193.107:0/171152731":"2026-02-20T23:46:25.777375+0000","10.20.193.107:0/3087895296":"2026-02-20T23:49:09.541423+0000","10.20.193.107:0/1222076271":"2026-02-20T23:47:21.604186+0000","10.20.193.107:0/420618425":"2026-02-20T23:52:03.555535+0000","10.20.193.107:0/1905443386":"2026-02-20T23:51:07.456890+0000","10.20.193.107:0/267882799":"2026-02-20T23:46:32.763563+0000","10.20.193.107:0/372312121":"2026-02-21T00:00:57.803126+0000","10.20.193.107:0/2682315323":"2026-02-20T23:46:49.452375+0000","10.20.193.107:0/2631924899":"2026-02-20T23:46:18.780502+0000","10.20.193.107:0/4282135197":"2026-02-20T23:51:47.551324+0000","10.20.193.107:0/880521052":"2026-02-20T23:27:08.066797+0000","10.20.193.107:0/4164259265":"2026-02-20T23:46:57.463343+0000","10.20.193.107:0/207494051":"2026-02-20T23:32:06.473308+0000","10.20.193.107:0/3134720608":"2026-02-20T23:27:58.060117+0000","10.20.193.107:0/3745545960":"2026-02-20T23:47:45.647954+0000","10.20.193.107:0/916353261":"2026-02-20T23:30:26.259588+0000","10.20.193.107:0/3819193199":"2026-02-20T23:45:33.632058+0000","10.20.193.107:0/1840450799":"2026-02-20T23:32:57.230807+0000","10.20.193.107:0/1179323197":"2026-02-20T23:56:17.530242+0000","10.20.193.107:0/1005054573":"2026-02-20T23:45:55.743376+0000","10.20.193.107:0/3523155574":"2026-02-20T23:28:48.723116+0000","10.20.193.107:0/829020688":"2026-02-20T23:47:37.666995+0000","10.20.193.107:0/2607914619":"2026-02-20T23:47:29.646974+0000","10.20.193.107:0/2828347503":"2026-02-20T23:54:48.969470+0000","10.20.193.107:0/3326442098":"2026-02-20T23:31:18.634486+0000","10.20.193.107:0/795169153":"2026-02-20T23:45:25.066478+0000","10.20.193.107:0/2947966439":"2026-02-20T23:48:21.430594+0000","10.20.193.107:0/4060359197":"2026-02-20T23:49:17.542712+0000","10.20.193.107:0/2345399195":"2026-02-20T23:45:41.622430+0000","10.20.193.107:0/259656318":"2026-02-20T23:50:02.779695+0000","10.20.193.107:0/1099066858":"2026-02-20T23:59:01.454369+0000","10.20.193.107:0/2882386397":"2026-02-20T23:47:58.775706+0000","10.20.193.107:0/1408528989":"2026-02-20T23:34:43.831351+0000","10.20.193.107:0/1266139039":"2026-02-20T23:48:05.776091+0000","10.20.193.107:0/512001260":"2026-02-20T23:49:33.598512+0000","10.20.193.107:0/623860268":"2026-02-20T23:52:11.807610+0000","10.20.193.107:0/2789114498":"2026-02-20T23:48:29.691764+0000","10.20.193.107:0/2288451591":"2026-02-20T23:33:50.744256+0000","10.20.193.107:0/2121489169":"2026-02-20T23:48:45.489262+0000","10.20.193.107:0/2742424328":"2026-02-20T23:45:49.647737+0000","10.20.193.107:0/60514943":"2026-02-20T23:48:37.474418+0000","10.20.193.107:0/4158780397":"2026-02-20T23:48:53.691140+0000","10.20.193.107:0/2759398604":"2026-02-20T23:51:15.453341+0000","10.20.193.107:0/1161542394":"2026-02-20T23:51:23.473614+0000","10.20.193.107:0/2884691340":"2026-02-20T23:51:55.580764+0000","10.20.193.107:0/2214326481":"2026-02-20T23:47:13.502090+0000","10.20.193.107:0/4099255251":"2026-02-20T23:49:49.654909+0000","10.20.193.107:0/1703422722":"2026-02-20T23:50:11.444875+0000","10.20.193.107:0/2653782187":"2026-02-20T23:49:25.594932+0000","10.20.193.107:0/2345388511":"2026-02-20T23:49:55.980340+0000","10.20.193.107:0/999079832":"2026-02-20T23:49:41.870082+0000","10.20.193.107:0/2829174269":"2026-02-20T23:50:27.680127+0000","10.20.193.107:0/236199311":"2026-02-20T23:50:35.673296+0000","10.20.193.107:0/134113142":"2026-02-20T23:50:43.665990+0000","10.20.193.107:0/3775237948":"2026-02-20T23:49:01.522647+0000","10.20.193.107:0/1626084429":"2026-02-20T23:50:51.645972+0000","10.20.193.107:0/3172672499":"2026-02-20T23:57:44.234853+0000","10.20.193.107:0/2961460377":"2026-02-20T23:50:59.456393+0000","10.20.193.107:0/1373644439":"2026-02-20T23:51:39.525146+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T23:01:05.532 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2026-02-20T23:01:05.533 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:05.608 INFO:teuthology.orchestra.run.trial107.stdout:{ 2026-02-20T23:01:05.608 INFO:teuthology.orchestra.run.trial107.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-20T23:01:05.608 INFO:teuthology.orchestra.run.trial107.stdout:} 2026-02-20T23:01:05.617 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2026-02-20T23:01:05.787 INFO:teuthology.orchestra.run.trial107.stderr:instructed osd(s) 0 to deep-scrub 2026-02-20T23:01:05.799 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2026-02-20T23:01:05.799 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:05.884 INFO:teuthology.orchestra.run.trial107.stdout:{ 2026-02-20T23:01:05.885 INFO:teuthology.orchestra.run.trial107.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-20T23:01:05.885 INFO:teuthology.orchestra.run.trial107.stdout:} 2026-02-20T23:01:05.894 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2026-02-20T23:01:06.053 INFO:teuthology.orchestra.run.trial107.stderr:instructed osd(s) 1 to deep-scrub 2026-02-20T23:01:06.065 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2026-02-20T23:01:06.065 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:06.143 INFO:teuthology.orchestra.run.trial107.stdout:{ 2026-02-20T23:01:06.143 INFO:teuthology.orchestra.run.trial107.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_degraded = '' osd_recovery_sleep_degraded_hdd = '' osd_recovery_sleep_degraded_hybrid = '' osd_recovery_sleep_degraded_ssd = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' " 2026-02-20T23:01:06.144 INFO:teuthology.orchestra.run.trial107.stdout:} 2026-02-20T23:01:06.152 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2026-02-20T23:01:06.305 INFO:teuthology.orchestra.run.trial107.stderr:instructed osd(s) 2 to deep-scrub 2026-02-20T23:01:06.317 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T23:01:06.468 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:06.469 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T23:01:06.478 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":1529,"stamp":"2026-02-20T23:01:05.742141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3445252,"kb_used_data":16980,"kb_used_omap":1407,"kb_used_meta":3426816,"kb_avail":279670268,"statfs":{"total":289910292480,"available":286382354432,"internally_reserved":0,"allocated":17387520,"data_stored":9125462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1441312,"internal_metadata":3509060064},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":17,"apply_latency_ms":17,"commit_latency_ns":17000000,"apply_latency_ns":17000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-536621110,"num_objects":-173,"num_object_clones":0,"num_object_copies":-346,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-173,"num_whiteouts":0,"num_read":-119090,"num_read_kb":-39715444,"num_write":-1286248,"num_write_kb":-486644537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":-16,"num_keys_recovered":0,"num_objects_omap":-18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.946915"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806029+0000","last_change":"2026-02-20T22:25:20.853750+0000","last_active":"2026-02-20T23:01:00.806029+0000","last_peered":"2026-02-20T23:01:00.806029+0000","last_clean":"2026-02-20T23:01:00.806029+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T23:01:00.806029+0000","last_undegraded":"2026-02-20T23:01:00.806029+0000","last_fullsized":"2026-02-20T23:01:00.806029+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:28:25.446423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":918,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.806924+0000","last_change":"2026-02-20T22:25:20.853533+0000","last_active":"2026-02-20T23:01:00.806924+0000","last_peered":"2026-02-20T23:01:00.806924+0000","last_clean":"2026-02-20T23:01:00.806924+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T23:01:00.806924+0000","last_undegraded":"2026-02-20T23:01:00.806924+0000","last_fullsized":"2026-02-20T23:01:00.806924+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:56:43.186123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.809121+0000","last_change":"2026-02-20T22:25:20.853536+0000","last_active":"2026-02-20T23:01:00.809121+0000","last_peered":"2026-02-20T23:01:00.809121+0000","last_clean":"2026-02-20T23:01:00.809121+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T23:01:00.809121+0000","last_undegraded":"2026-02-20T23:01:00.809121+0000","last_fullsized":"2026-02-20T23:01:00.809121+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:57:45.351507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":919,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.800318+0000","last_change":"2026-02-20T22:25:20.853322+0000","last_active":"2026-02-20T23:01:00.800318+0000","last_peered":"2026-02-20T23:01:00.800318+0000","last_clean":"2026-02-20T23:01:00.800318+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T23:01:00.800318+0000","last_undegraded":"2026-02-20T23:01:00.800318+0000","last_fullsized":"2026-02-20T23:01:00.800318+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:16:23.680830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":921,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:03.506454+0000","last_change":"2026-02-20T22:25:20.851616+0000","last_active":"2026-02-20T23:01:03.506454+0000","last_peered":"2026-02-20T23:01:03.506454+0000","last_clean":"2026-02-20T23:01:03.506454+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T23:01:03.506454+0000","last_undegraded":"2026-02-20T23:01:03.506454+0000","last_fullsized":"2026-02-20T23:01:03.506454+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:46:52.499153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":921,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.807656+0000","last_change":"2026-02-20T22:25:20.848571+0000","last_active":"2026-02-20T23:01:02.807656+0000","last_peered":"2026-02-20T23:01:02.807656+0000","last_clean":"2026-02-20T23:01:02.807656+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T23:01:02.807656+0000","last_undegraded":"2026-02-20T23:01:02.807656+0000","last_fullsized":"2026-02-20T23:01:02.807656+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T02:30:13.829834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":921,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.806902+0000","last_change":"2026-02-20T22:25:20.848547+0000","last_active":"2026-02-20T23:01:02.806902+0000","last_peered":"2026-02-20T23:01:02.806902+0000","last_clean":"2026-02-20T23:01:02.806902+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T23:01:02.806902+0000","last_undegraded":"2026-02-20T23:01:02.806902+0000","last_fullsized":"2026-02-20T23:01:02.806902+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:13:29.729280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":920,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.807135+0000","last_change":"2026-02-20T22:25:20.853349+0000","last_active":"2026-02-20T23:01:00.807135+0000","last_peered":"2026-02-20T23:01:00.807135+0000","last_clean":"2026-02-20T23:01:00.807135+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T23:01:00.807135+0000","last_undegraded":"2026-02-20T23:01:00.807135+0000","last_fullsized":"2026-02-20T23:01:00.807135+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:17.838849+0000","last_clean_scrub_stamp":"2026-02-20T22:25:17.838849+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:45:40.392540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1008,"reported_epoch":455,"state":"active+clean","last_fresh":"2026-02-20T23:01:00.798989+0000","last_change":"2026-02-20T22:25:17.165550+0000","last_active":"2026-02-20T23:01:00.798989+0000","last_peered":"2026-02-20T23:01:00.798989+0000","last_clean":"2026-02-20T23:01:00.798989+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T23:01:00.798989+0000","last_undegraded":"2026-02-20T23:01:00.798989+0000","last_fullsized":"2026-02-20T23:01:00.798989+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:25:15.833567+0000","last_clean_scrub_stamp":"2026-02-20T22:25:15.833567+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:42:35.804057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738800,"num_pgs":99,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1185116,"kb_used_data":5836,"kb_used_omap":122,"kb_used_meta":1179141,"kb_avail":93186724,"statfs":{"total":96636764160,"available":95423205376,"internally_reserved":0,"allocated":5976064,"data_stored":3229455,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":125580,"internal_metadata":1207440756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738802,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289712,"kb_used_data":5280,"kb_used_omap":1074,"kb_used_meta":1283341,"kb_avail":93082128,"statfs":{"total":96636764160,"available":95316099072,"internally_reserved":0,"allocated":5406720,"data_stored":2652810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1100691,"internal_metadata":1314141293},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738801,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970424,"kb_used_data":5864,"kb_used_omap":210,"kb_used_meta":964333,"kb_avail":93401416,"statfs":{"total":96636764160,"available":95643049984,"internally_reserved":0,"allocated":6004736,"data_stored":3243197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":215041,"internal_metadata":987478015},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:06.481 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-20T22:25:17.838849+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=17, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-20T22:25:15.833567+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=25, tm_sec=15, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=4, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:06.482 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-20T23:01:26.483 DEBUG:teuthology.orchestra.run.trial107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2026-02-20T23:01:26.634 INFO:teuthology.orchestra.run.trial107.stdout: 2026-02-20T23:01:26.634 INFO:teuthology.orchestra.run.trial107.stderr:dumped all 2026-02-20T23:01:26.644 INFO:teuthology.orchestra.run.trial107.stdout:{"pg_ready":true,"pg_map":{"version":1539,"stamp":"2026-02-20T23:01:25.743608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":3463328,"kb_used_data":17008,"kb_used_omap":1410,"kb_used_meta":3444861,"kb_avail":279652192,"statfs":{"total":289910292480,"available":286363844608,"internally_reserved":0,"allocated":17416192,"data_stored":9139204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1444628,"internal_metadata":3527537900},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000822"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":928,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:07.110520+0000","last_change":"2026-02-20T23:01:07.110520+0000","last_active":"2026-02-20T23:01:07.110520+0000","last_peered":"2026-02-20T23:01:07.110520+0000","last_clean":"2026-02-20T23:01:07.110520+0000","last_became_active":"2026-02-20T22:25:18.848125+0000","last_became_peered":"2026-02-20T22:25:18.848125+0000","last_unstale":"2026-02-20T23:01:07.110520+0000","last_undegraded":"2026-02-20T23:01:07.110520+0000","last_fullsized":"2026-02-20T23:01:07.110520+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:07.110418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:07.110418+0000","last_clean_scrub_stamp":"2026-02-20T23:01:07.110418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:57:36.279270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010710769999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":928,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:08.084054+0000","last_change":"2026-02-20T23:01:08.084054+0000","last_active":"2026-02-20T23:01:08.084054+0000","last_peered":"2026-02-20T23:01:08.084054+0000","last_clean":"2026-02-20T23:01:08.084054+0000","last_became_active":"2026-02-20T22:25:18.848262+0000","last_became_peered":"2026-02-20T22:25:18.848262+0000","last_unstale":"2026-02-20T23:01:08.084054+0000","last_undegraded":"2026-02-20T23:01:08.084054+0000","last_fullsized":"2026-02-20T23:01:08.084054+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:08.083972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:08.083972+0000","last_clean_scrub_stamp":"2026-02-20T23:01:08.083972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T10:15:34.390229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086154799999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":929,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:10.162926+0000","last_change":"2026-02-20T23:01:10.162926+0000","last_active":"2026-02-20T23:01:10.162926+0000","last_peered":"2026-02-20T23:01:10.162926+0000","last_clean":"2026-02-20T23:01:10.162926+0000","last_became_active":"2026-02-20T22:25:18.848468+0000","last_became_peered":"2026-02-20T22:25:18.848468+0000","last_unstale":"2026-02-20T23:01:10.162926+0000","last_undegraded":"2026-02-20T23:01:10.162926+0000","last_fullsized":"2026-02-20T23:01:10.162926+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:10.162859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:10.162859+0000","last_clean_scrub_stamp":"2026-02-20T23:01:10.162859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:29:22.267872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094832900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":929,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:11.185802+0000","last_change":"2026-02-20T23:01:11.185802+0000","last_active":"2026-02-20T23:01:11.185802+0000","last_peered":"2026-02-20T23:01:11.185802+0000","last_clean":"2026-02-20T23:01:11.185802+0000","last_became_active":"2026-02-20T22:25:18.848626+0000","last_became_peered":"2026-02-20T22:25:18.848626+0000","last_unstale":"2026-02-20T23:01:11.185802+0000","last_undegraded":"2026-02-20T23:01:11.185802+0000","last_fullsized":"2026-02-20T23:01:11.185802+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:11.185715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:11.185715+0000","last_clean_scrub_stamp":"2026-02-20T23:01:11.185715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T03:35:25.266219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00066991200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"456'3","reported_seq":930,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:06.161261+0000","last_change":"2026-02-20T23:01:06.161261+0000","last_active":"2026-02-20T23:01:06.161261+0000","last_peered":"2026-02-20T23:01:06.161261+0000","last_clean":"2026-02-20T23:01:06.161261+0000","last_became_active":"2026-02-20T22:25:18.847703+0000","last_became_peered":"2026-02-20T22:25:18.847703+0000","last_unstale":"2026-02-20T23:01:06.161261+0000","last_undegraded":"2026-02-20T23:01:06.161261+0000","last_fullsized":"2026-02-20T23:01:06.161261+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'3","last_scrub_stamp":"2026-02-20T23:01:06.161238+0000","last_deep_scrub":"456'3","last_deep_scrub_stamp":"2026-02-20T23:01:06.161238+0000","last_clean_scrub_stamp":"2026-02-20T23:01:06.161238+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:43:58.751909+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00092232100000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":929,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:07.693844+0000","last_change":"2026-02-20T23:01:07.693844+0000","last_active":"2026-02-20T23:01:07.693844+0000","last_peered":"2026-02-20T23:01:07.693844+0000","last_clean":"2026-02-20T23:01:07.693844+0000","last_became_active":"2026-02-20T22:25:18.848227+0000","last_became_peered":"2026-02-20T22:25:18.848227+0000","last_unstale":"2026-02-20T23:01:07.693844+0000","last_undegraded":"2026-02-20T23:01:07.693844+0000","last_fullsized":"2026-02-20T23:01:07.693844+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:07.693739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:07.693739+0000","last_clean_scrub_stamp":"2026-02-20T23:01:07.693739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:37:44.287577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037505500000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":930,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:06.691367+0000","last_change":"2026-02-20T23:01:06.691279+0000","last_active":"2026-02-20T23:01:06.691367+0000","last_peered":"2026-02-20T23:01:06.691367+0000","last_clean":"2026-02-20T23:01:06.691367+0000","last_became_active":"2026-02-20T22:25:18.847857+0000","last_became_peered":"2026-02-20T22:25:18.847857+0000","last_unstale":"2026-02-20T23:01:06.691367+0000","last_undegraded":"2026-02-20T23:01:06.691367+0000","last_fullsized":"2026-02-20T23:01:06.691367+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:06.691163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:06.691163+0000","last_clean_scrub_stamp":"2026-02-20T23:01:06.691163+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T10:21:14.240107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000360298,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"456'2","reported_seq":931,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:09.126799+0000","last_change":"2026-02-20T23:01:09.126799+0000","last_active":"2026-02-20T23:01:09.126799+0000","last_peered":"2026-02-20T23:01:09.126799+0000","last_clean":"2026-02-20T23:01:09.126799+0000","last_became_active":"2026-02-20T22:25:18.847541+0000","last_became_peered":"2026-02-20T22:25:18.847541+0000","last_unstale":"2026-02-20T23:01:09.126799+0000","last_undegraded":"2026-02-20T23:01:09.126799+0000","last_fullsized":"2026-02-20T23:01:09.126799+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'2","last_scrub_stamp":"2026-02-20T23:01:09.126755+0000","last_deep_scrub":"456'2","last_deep_scrub_stamp":"2026-02-20T23:01:09.126755+0000","last_clean_scrub_stamp":"2026-02-20T23:01:09.126755+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:10:05.253588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075348799999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"456'74","reported_seq":1020,"reported_epoch":456,"state":"active+clean","last_fresh":"2026-02-20T23:01:06.075467+0000","last_change":"2026-02-20T23:01:06.075467+0000","last_active":"2026-02-20T23:01:06.075467+0000","last_peered":"2026-02-20T23:01:06.075467+0000","last_clean":"2026-02-20T23:01:06.075467+0000","last_became_active":"2026-02-20T22:25:17.164967+0000","last_became_peered":"2026-02-20T22:25:17.164967+0000","last_unstale":"2026-02-20T23:01:06.075467+0000","last_undegraded":"2026-02-20T23:01:06.075467+0000","last_fullsized":"2026-02-20T23:01:06.075467+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'74","last_scrub_stamp":"2026-02-20T23:01:06.075415+0000","last_deep_scrub":"456'74","last_deep_scrub_stamp":"2026-02-20T23:01:06.075415+0000","last_clean_scrub_stamp":"2026-02-20T23:01:06.075415+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:18:10.245938+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":1,"up_from":8,"seq":34359738804,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203192,"kb_used_data":5864,"kb_used_omap":125,"kb_used_meta":1197186,"kb_avail":93168648,"statfs":{"total":96636764160,"available":95404695552,"internally_reserved":0,"allocated":6004736,"data_stored":3243197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":128896,"internal_metadata":1225918592},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":8,"seq":34359738806,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1289712,"kb_used_data":5280,"kb_used_omap":1074,"kb_used_meta":1283341,"kb_avail":93082128,"statfs":{"total":96636764160,"available":95316099072,"internally_reserved":0,"allocated":5406720,"data_stored":2652810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1100691,"internal_metadata":1314141293},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738805,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":970424,"kb_used_data":5864,"kb_used_omap":210,"kb_used_meta":964333,"kb_avail":93401416,"statfs":{"total":96636764160,"available":95643049984,"internally_reserved":0,"allocated":6004736,"data_stored":3243197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":215041,"internal_metadata":987478015},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:26.646 DEBUG:teuthology.orchestra.run.trial107:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2026-02-20T23:01:26.815 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-20T23:01:26.815 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-20T23:01:26.815 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2026-02-20T23:01:26.815 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:29.779 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2026-02-20T23:01:32.916 INFO:tasks.ceph.cluster2.osd.0:Stopped 2026-02-20T23:01:32.917 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2026-02-20T23:01:32.917 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:35.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2026-02-20T23:01:35.082 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2026-02-20T23:01:39.018 INFO:tasks.ceph.cluster2.osd.1:Stopped 2026-02-20T23:01:39.018 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2026-02-20T23:01:39.018 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:40.283 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2026-02-20T23:01:40.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2026-02-20T23:01:45.120 INFO:tasks.ceph.cluster2.osd.2:Stopped 2026-02-20T23:01:45.120 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-20T23:01:45.121 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2026-02-20T23:01:45.121 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:45.178 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2026-02-20T23:01:45.178 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-20T23:01:45.179 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2026-02-20T23:01:45.179 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:01:45.197 INFO:tasks.ceph.cluster2.mon.a:Stopped 2026-02-20T23:01:45.198 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-20T23:01:45.198 DEBUG:teuthology.orchestra.run.trial107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-20T23:01:45.248 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T23:01:45.248 DEBUG:teuthology.orchestra.run.trial107:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2026-02-20T23:01:45.284 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2026-02-20T23:01:45.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T23:01:45.390 DEBUG:teuthology.orchestra.run.trial107:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2026-02-20T23:01:45.472 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@trial107.front.sepia.ceph.com 2026-02-20T23:01:45.472 DEBUG:teuthology.orchestra.run.trial107:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2026-02-20T23:01:45.548 INFO:tasks.ceph:Archiving mon data... 2026-02-20T23:01:45.551 DEBUG:teuthology.misc:Transferring archived files from trial107:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/data/cluster2.mon.a.tgz 2026-02-20T23:01:45.551 DEBUG:teuthology.orchestra.run.trial107:> mktemp 2026-02-20T23:01:45.557 INFO:teuthology.orchestra.run.trial107.stdout:/tmp/tmp.uMY3UYsQro 2026-02-20T23:01:45.557 DEBUG:teuthology.orchestra.run.trial107:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.uMY3UYsQro 2026-02-20T23:01:45.667 DEBUG:teuthology.orchestra.run.trial107:> sudo chmod 0666 /tmp/tmp.uMY3UYsQro 2026-02-20T23:01:45.727 DEBUG:teuthology.orchestra.remote:trial107:/tmp/tmp.uMY3UYsQro is 489KB 2026-02-20T23:01:45.787 DEBUG:teuthology.orchestra.run.trial107:> rm -fr /tmp/tmp.uMY3UYsQro 2026-02-20T23:01:45.792 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-20T23:01:45.792 DEBUG:teuthology.orchestra.run.trial032:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T23:01:45.794 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T23:01:45.842 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2026-02-20T23:01:45.861 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2026-02-20T23:01:45.861 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T23:01:46.021 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:01:46.022 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T23:01:46.031 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2026-02-20T23:01:45.131020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":291496,"kb_used_data":12280,"kb_used_omap":8111,"kb_used_meta":271056,"kb_avail":282824024,"statfs":{"total":289910292480,"available":289611800576,"internally_reserved":0,"allocated":12574720,"data_stored":7280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8306057,"internal_metadata":277561975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353096+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T23:01:02.353096+0000","last_peered":"2026-02-20T23:01:02.353096+0000","last_clean":"2026-02-20T23:01:02.353096+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T23:01:02.353096+0000","last_undegraded":"2026-02-20T23:01:02.353096+0000","last_fullsized":"2026-02-20T23:01:02.353096+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353111+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T23:01:02.353111+0000","last_peered":"2026-02-20T23:01:02.353111+0000","last_clean":"2026-02-20T23:01:02.353111+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T23:01:02.353111+0000","last_undegraded":"2026-02-20T23:01:02.353111+0000","last_fullsized":"2026-02-20T23:01:02.353111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353838+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T23:01:02.353838+0000","last_peered":"2026-02-20T23:01:02.353838+0000","last_clean":"2026-02-20T23:01:02.353838+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T23:01:02.353838+0000","last_undegraded":"2026-02-20T23:01:02.353838+0000","last_fullsized":"2026-02-20T23:01:02.353838+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.351624+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T23:01:02.351624+0000","last_peered":"2026-02-20T23:01:02.351624+0000","last_clean":"2026-02-20T23:01:02.351624+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T23:01:02.351624+0000","last_undegraded":"2026-02-20T23:01:02.351624+0000","last_fullsized":"2026-02-20T23:01:02.351624+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.248026+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T23:01:02.248026+0000","last_peered":"2026-02-20T23:01:02.248026+0000","last_clean":"2026-02-20T23:01:02.248026+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T23:01:02.248026+0000","last_undegraded":"2026-02-20T23:01:02.248026+0000","last_fullsized":"2026-02-20T23:01:02.248026+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823164+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T23:01:01.823164+0000","last_peered":"2026-02-20T23:01:01.823164+0000","last_clean":"2026-02-20T23:01:01.823164+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T23:01:01.823164+0000","last_undegraded":"2026-02-20T23:01:01.823164+0000","last_fullsized":"2026-02-20T23:01:01.823164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823173+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T23:01:01.823173+0000","last_peered":"2026-02-20T23:01:01.823173+0000","last_clean":"2026-02-20T23:01:01.823173+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T23:01:01.823173+0000","last_undegraded":"2026-02-20T23:01:01.823173+0000","last_fullsized":"2026-02-20T23:01:01.823173+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.352097+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T23:01:02.352097+0000","last_peered":"2026-02-20T23:01:02.352097+0000","last_clean":"2026-02-20T23:01:02.352097+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T23:01:02.352097+0000","last_undegraded":"2026-02-20T23:01:02.352097+0000","last_fullsized":"2026-02-20T23:01:02.352097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1020,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.349319+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T23:01:02.349319+0000","last_peered":"2026-02-20T23:01:02.349319+0000","last_clean":"2026-02-20T23:01:02.349319+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T23:01:02.349319+0000","last_undegraded":"2026-02-20T23:01:02.349319+0000","last_fullsized":"2026-02-20T23:01:02.349319+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":3704,"kb_used_omap":2983,"kb_used_meta":78040,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":3792896,"data_stored":2033263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3055400,"internal_metadata":79913176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85328,"kb_used_data":4288,"kb_used_omap":1999,"kb_used_meta":79024,"kb_avail":94286512,"statfs":{"total":96636764160,"available":96549388288,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2047842,"internal_metadata":80920734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121424,"kb_used_data":4288,"kb_used_omap":3127,"kb_used_meta":113992,"kb_avail":94250416,"statfs":{"total":96636764160,"available":96512425984,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202815,"internal_metadata":116728065},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:46.033 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T23:01:46.192 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:01:46.192 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T23:01:46.204 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2026-02-20T23:01:45.131020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":291496,"kb_used_data":12280,"kb_used_omap":8111,"kb_used_meta":271056,"kb_avail":282824024,"statfs":{"total":289910292480,"available":289611800576,"internally_reserved":0,"allocated":12574720,"data_stored":7280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8306057,"internal_metadata":277561975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353096+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T23:01:02.353096+0000","last_peered":"2026-02-20T23:01:02.353096+0000","last_clean":"2026-02-20T23:01:02.353096+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T23:01:02.353096+0000","last_undegraded":"2026-02-20T23:01:02.353096+0000","last_fullsized":"2026-02-20T23:01:02.353096+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353111+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T23:01:02.353111+0000","last_peered":"2026-02-20T23:01:02.353111+0000","last_clean":"2026-02-20T23:01:02.353111+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T23:01:02.353111+0000","last_undegraded":"2026-02-20T23:01:02.353111+0000","last_fullsized":"2026-02-20T23:01:02.353111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353838+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T23:01:02.353838+0000","last_peered":"2026-02-20T23:01:02.353838+0000","last_clean":"2026-02-20T23:01:02.353838+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T23:01:02.353838+0000","last_undegraded":"2026-02-20T23:01:02.353838+0000","last_fullsized":"2026-02-20T23:01:02.353838+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.351624+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T23:01:02.351624+0000","last_peered":"2026-02-20T23:01:02.351624+0000","last_clean":"2026-02-20T23:01:02.351624+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T23:01:02.351624+0000","last_undegraded":"2026-02-20T23:01:02.351624+0000","last_fullsized":"2026-02-20T23:01:02.351624+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.248026+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T23:01:02.248026+0000","last_peered":"2026-02-20T23:01:02.248026+0000","last_clean":"2026-02-20T23:01:02.248026+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T23:01:02.248026+0000","last_undegraded":"2026-02-20T23:01:02.248026+0000","last_fullsized":"2026-02-20T23:01:02.248026+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823164+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T23:01:01.823164+0000","last_peered":"2026-02-20T23:01:01.823164+0000","last_clean":"2026-02-20T23:01:01.823164+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T23:01:01.823164+0000","last_undegraded":"2026-02-20T23:01:01.823164+0000","last_fullsized":"2026-02-20T23:01:01.823164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823173+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T23:01:01.823173+0000","last_peered":"2026-02-20T23:01:01.823173+0000","last_clean":"2026-02-20T23:01:01.823173+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T23:01:01.823173+0000","last_undegraded":"2026-02-20T23:01:01.823173+0000","last_fullsized":"2026-02-20T23:01:01.823173+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.352097+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T23:01:02.352097+0000","last_peered":"2026-02-20T23:01:02.352097+0000","last_clean":"2026-02-20T23:01:02.352097+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T23:01:02.352097+0000","last_undegraded":"2026-02-20T23:01:02.352097+0000","last_fullsized":"2026-02-20T23:01:02.352097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1020,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.349319+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T23:01:02.349319+0000","last_peered":"2026-02-20T23:01:02.349319+0000","last_clean":"2026-02-20T23:01:02.349319+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T23:01:02.349319+0000","last_undegraded":"2026-02-20T23:01:02.349319+0000","last_fullsized":"2026-02-20T23:01:02.349319+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":3704,"kb_used_omap":2983,"kb_used_meta":78040,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":3792896,"data_stored":2033263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3055400,"internal_metadata":79913176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85328,"kb_used_data":4288,"kb_used_omap":1999,"kb_used_meta":79024,"kb_avail":94286512,"statfs":{"total":96636764160,"available":96549388288,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2047842,"internal_metadata":80920734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121424,"kb_used_data":4288,"kb_used_omap":3127,"kb_used_meta":113992,"kb_avail":94250416,"statfs":{"total":96636764160,"available":96512425984,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202815,"internal_metadata":116728065},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:46.205 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2026-02-20T23:01:46.206 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T23:01:46.366 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:01:46.367 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T23:01:46.376 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":1566,"stamp":"2026-02-20T23:01:45.131020+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":291496,"kb_used_data":12280,"kb_used_omap":8111,"kb_used_meta":271056,"kb_avail":282824024,"statfs":{"total":289910292480,"available":289611800576,"internally_reserved":0,"allocated":12574720,"data_stored":7280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8306057,"internal_metadata":277561975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000945"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353096+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T23:01:02.353096+0000","last_peered":"2026-02-20T23:01:02.353096+0000","last_clean":"2026-02-20T23:01:02.353096+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T23:01:02.353096+0000","last_undegraded":"2026-02-20T23:01:02.353096+0000","last_fullsized":"2026-02-20T23:01:02.353096+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":929,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353111+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T23:01:02.353111+0000","last_peered":"2026-02-20T23:01:02.353111+0000","last_clean":"2026-02-20T23:01:02.353111+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T23:01:02.353111+0000","last_undegraded":"2026-02-20T23:01:02.353111+0000","last_fullsized":"2026-02-20T23:01:02.353111+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353838+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T23:01:02.353838+0000","last_peered":"2026-02-20T23:01:02.353838+0000","last_clean":"2026-02-20T23:01:02.353838+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T23:01:02.353838+0000","last_undegraded":"2026-02-20T23:01:02.353838+0000","last_fullsized":"2026-02-20T23:01:02.353838+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.351624+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T23:01:02.351624+0000","last_peered":"2026-02-20T23:01:02.351624+0000","last_clean":"2026-02-20T23:01:02.351624+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T23:01:02.351624+0000","last_undegraded":"2026-02-20T23:01:02.351624+0000","last_fullsized":"2026-02-20T23:01:02.351624+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.248026+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T23:01:02.248026+0000","last_peered":"2026-02-20T23:01:02.248026+0000","last_clean":"2026-02-20T23:01:02.248026+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T23:01:02.248026+0000","last_undegraded":"2026-02-20T23:01:02.248026+0000","last_fullsized":"2026-02-20T23:01:02.248026+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823164+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T23:01:01.823164+0000","last_peered":"2026-02-20T23:01:01.823164+0000","last_clean":"2026-02-20T23:01:01.823164+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T23:01:01.823164+0000","last_undegraded":"2026-02-20T23:01:01.823164+0000","last_fullsized":"2026-02-20T23:01:01.823164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823173+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T23:01:01.823173+0000","last_peered":"2026-02-20T23:01:01.823173+0000","last_clean":"2026-02-20T23:01:01.823173+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T23:01:01.823173+0000","last_undegraded":"2026-02-20T23:01:01.823173+0000","last_fullsized":"2026-02-20T23:01:01.823173+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.352097+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T23:01:02.352097+0000","last_peered":"2026-02-20T23:01:02.352097+0000","last_clean":"2026-02-20T23:01:02.352097+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T23:01:02.352097+0000","last_undegraded":"2026-02-20T23:01:02.352097+0000","last_fullsized":"2026-02-20T23:01:02.352097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1020,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.349319+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T23:01:02.349319+0000","last_peered":"2026-02-20T23:01:02.349319+0000","last_clean":"2026-02-20T23:01:02.349319+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T23:01:02.349319+0000","last_undegraded":"2026-02-20T23:01:02.349319+0000","last_fullsized":"2026-02-20T23:01:02.349319+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738813,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":3704,"kb_used_omap":2983,"kb_used_meta":78040,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":3792896,"data_stored":2033263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3055400,"internal_metadata":79913176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738813,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85328,"kb_used_data":4288,"kb_used_omap":1999,"kb_used_meta":79024,"kb_avail":94286512,"statfs":{"total":96636764160,"available":96549388288,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2047842,"internal_metadata":80920734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738812,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121424,"kb_used_data":4288,"kb_used_omap":3127,"kb_used_meta":113992,"kb_avail":94250416,"statfs":{"total":96636764160,"available":96512425984,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202815,"internal_metadata":116728065},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:46.378 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2026-02-20T23:01:46.529 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:01:46.529 INFO:teuthology.orchestra.run.trial032.stdout:{"epoch":461,"fsid":"d1a0e502-1b23-4a49-a64f-9ce85d3de0ff","created":"2026-02-20T22:24:46.656466+0000","modified":"2026-02-20T23:01:01.816280+0000","last_up_change":"2026-02-20T22:24:51.859134+0000","last_in_change":"2026-02-20T22:24:47.280068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"tentacle","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2026-02-20T22:24:52.849512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2026-02-20T22:24:54.413552+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"nonprimary_shards":"{}","options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9ca3162b-8d73-4cac-a717-cf31c5b51d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6800","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6801","nonce":3011236143}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6802","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6803","nonce":3011236143}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6806","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6807","nonce":3011236143}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6804","nonce":3011236143},{"type":"v1","addr":"10.20.193.32:6805","nonce":3011236143}]},"public_addr":"10.20.193.32:6801/3011236143","cluster_addr":"10.20.193.32:6803/3011236143","heartbeat_back_addr":"10.20.193.32:6807/3011236143","heartbeat_front_addr":"10.20.193.32:6805/3011236143","state":["exists","up"]},{"osd":1,"uuid":"65fbbb8d-e0b9-4e23-8050-11c3774a6bd9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6809","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6812","nonce":1911326603}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6815","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6817","nonce":1911326603}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6822","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6823","nonce":1911326603}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6820","nonce":1911326603},{"type":"v1","addr":"10.20.193.32:6821","nonce":1911326603}]},"public_addr":"10.20.193.32:6812/1911326603","cluster_addr":"10.20.193.32:6817/1911326603","heartbeat_back_addr":"10.20.193.32:6823/1911326603","heartbeat_front_addr":"10.20.193.32:6821/1911326603","state":["exists","up"]},{"osd":2,"uuid":"02f1be14-e4b6-4854-8214-b3c8094dc992","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6808","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6810","nonce":2685558554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6811","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6813","nonce":2685558554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6818","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6819","nonce":2685558554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"10.20.193.32:6814","nonce":2685558554},{"type":"v1","addr":"10.20.193.32:6816","nonce":2685558554}]},"public_addr":"10.20.193.32:6810/2685558554","cluster_addr":"10.20.193.32:6813/2685558554","heartbeat_back_addr":"10.20.193.32:6819/2685558554","heartbeat_front_addr":"10.20.193.32:6816/2685558554","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:24:50.120221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:24:50.071223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4544132024016699391,"old_weight":0,"last_purged_snaps_scrub":"2026-02-20T22:24:50.139756+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"isa","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2026-02-20T23:01:47.543 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2026-02-20T23:01:47.543 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:47.631 INFO:teuthology.orchestra.run.trial032.stdout:{ 2026-02-20T23:01:47.631 INFO:teuthology.orchestra.run.trial032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-20T23:01:47.631 INFO:teuthology.orchestra.run.trial032.stdout:} 2026-02-20T23:01:47.640 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2026-02-20T23:01:47.795 INFO:teuthology.orchestra.run.trial032.stderr:instructed osd(s) 0 to deep-scrub 2026-02-20T23:01:47.806 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2026-02-20T23:01:47.807 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:47.881 INFO:teuthology.orchestra.run.trial032.stdout:{ 2026-02-20T23:01:47.882 INFO:teuthology.orchestra.run.trial032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-20T23:01:47.882 INFO:teuthology.orchestra.run.trial032.stdout:} 2026-02-20T23:01:47.891 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2026-02-20T23:01:48.038 INFO:teuthology.orchestra.run.trial032.stderr:instructed osd(s) 1 to deep-scrub 2026-02-20T23:01:48.051 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2026-02-20T23:01:48.051 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2026-02-20T23:01:48.136 INFO:teuthology.orchestra.run.trial032.stdout:{ 2026-02-20T23:01:48.136 INFO:teuthology.orchestra.run.trial032.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2026-02-20T23:01:48.136 INFO:teuthology.orchestra.run.trial032.stdout:} 2026-02-20T23:01:48.144 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2026-02-20T23:01:48.292 INFO:teuthology.orchestra.run.trial032.stderr:instructed osd(s) 2 to deep-scrub 2026-02-20T23:01:48.303 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T23:01:48.452 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:01:48.453 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T23:01:48.464 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":1567,"stamp":"2026-02-20T23:01:47.131147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":291496,"kb_used_data":12280,"kb_used_omap":8111,"kb_used_meta":271056,"kb_avail":282824024,"statfs":{"total":289910292480,"available":289611800576,"internally_reserved":0,"allocated":12574720,"data_stored":7280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8306057,"internal_metadata":277561975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000897"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353152+0000","last_change":"2026-02-20T22:24:57.881082+0000","last_active":"2026-02-20T23:01:02.353152+0000","last_peered":"2026-02-20T23:01:02.353152+0000","last_clean":"2026-02-20T23:01:02.353152+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T23:01:02.353152+0000","last_undegraded":"2026-02-20T23:01:02.353152+0000","last_fullsized":"2026-02-20T23:01:02.353152+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T06:20:38.540322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353162+0000","last_change":"2026-02-20T22:24:57.880646+0000","last_active":"2026-02-20T23:01:02.353162+0000","last_peered":"2026-02-20T23:01:02.353162+0000","last_clean":"2026-02-20T23:01:02.353162+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T23:01:02.353162+0000","last_undegraded":"2026-02-20T23:01:02.353162+0000","last_fullsized":"2026-02-20T23:01:02.353162+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:00:47.412634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.353838+0000","last_change":"2026-02-20T22:24:57.880796+0000","last_active":"2026-02-20T23:01:02.353838+0000","last_peered":"2026-02-20T23:01:02.353838+0000","last_clean":"2026-02-20T23:01:02.353838+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T23:01:02.353838+0000","last_undegraded":"2026-02-20T23:01:02.353838+0000","last_fullsized":"2026-02-20T23:01:02.353838+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T00:27:16.354288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":930,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.351624+0000","last_change":"2026-02-20T22:24:57.880908+0000","last_active":"2026-02-20T23:01:02.351624+0000","last_peered":"2026-02-20T23:01:02.351624+0000","last_clean":"2026-02-20T23:01:02.351624+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T23:01:02.351624+0000","last_undegraded":"2026-02-20T23:01:02.351624+0000","last_fullsized":"2026-02-20T23:01:02.351624+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:11:09.944711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"14'2","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.248026+0000","last_change":"2026-02-20T22:24:57.881920+0000","last_active":"2026-02-20T23:01:02.248026+0000","last_peered":"2026-02-20T23:01:02.248026+0000","last_clean":"2026-02-20T23:01:02.248026+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T23:01:02.248026+0000","last_undegraded":"2026-02-20T23:01:02.248026+0000","last_fullsized":"2026-02-20T23:01:02.248026+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:18:43.836884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823164+0000","last_change":"2026-02-20T22:24:58.151557+0000","last_active":"2026-02-20T23:01:01.823164+0000","last_peered":"2026-02-20T23:01:01.823164+0000","last_clean":"2026-02-20T23:01:01.823164+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T23:01:01.823164+0000","last_undegraded":"2026-02-20T23:01:01.823164+0000","last_fullsized":"2026-02-20T23:01:01.823164+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:35:06.722353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":932,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:01.823173+0000","last_change":"2026-02-20T22:24:58.151512+0000","last_active":"2026-02-20T23:01:01.823173+0000","last_peered":"2026-02-20T23:01:01.823173+0000","last_clean":"2026-02-20T23:01:01.823173+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T23:01:01.823173+0000","last_undegraded":"2026-02-20T23:01:01.823173+0000","last_fullsized":"2026-02-20T23:01:01.823173+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:25:32.759780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"12'1","reported_seq":931,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.352097+0000","last_change":"2026-02-20T22:24:57.880556+0000","last_active":"2026-02-20T23:01:02.352097+0000","last_peered":"2026-02-20T23:01:02.352097+0000","last_clean":"2026-02-20T23:01:02.352097+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T23:01:02.352097+0000","last_undegraded":"2026-02-20T23:01:02.352097+0000","last_fullsized":"2026-02-20T23:01:02.352097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:54.868496+0000","last_clean_scrub_stamp":"2026-02-20T22:24:54.868496+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:24:30.452559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'72","reported_seq":1020,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:02.349319+0000","last_change":"2026-02-20T22:24:53.869939+0000","last_active":"2026-02-20T23:01:02.349319+0000","last_peered":"2026-02-20T23:01:02.349319+0000","last_clean":"2026-02-20T23:01:02.349319+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T23:01:02.349319+0000","last_undegraded":"2026-02-20T23:01:02.349319+0000","last_fullsized":"2026-02-20T23:01:02.349319+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T22:24:52.865061+0000","last_clean_scrub_stamp":"2026-02-20T22:24:52.865061+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:40:00.368516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738814,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":3704,"kb_used_omap":2983,"kb_used_meta":78040,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":3792896,"data_stored":2033263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3055400,"internal_metadata":79913176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738814,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85328,"kb_used_data":4288,"kb_used_omap":1999,"kb_used_meta":79024,"kb_avail":94286512,"statfs":{"total":96636764160,"available":96549388288,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2047842,"internal_metadata":80920734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738813,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121424,"kb_used_data":4288,"kb_used_omap":3127,"kb_used_meta":113992,"kb_avail":94250416,"statfs":{"total":96636764160,"available":96512425984,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202815,"internal_metadata":116728065},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:01:48.464 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2026-02-20T22:24:54.868496+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=54, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2026-02-20T22:24:52.865061+0000 time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=22, tm_min=24, tm_sec=52, tm_wday=4, tm_yday=51, tm_isdst=-1) <= time.struct_time(tm_year=2026, tm_mon=2, tm_mday=20, tm_hour=23, tm_min=1, tm_sec=46, tm_wday=4, tm_yday=51, tm_isdst=0) 2026-02-20T23:01:48.465 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2026-02-20T23:02:08.466 DEBUG:teuthology.orchestra.run.trial032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2026-02-20T23:02:08.624 INFO:teuthology.orchestra.run.trial032.stdout: 2026-02-20T23:02:08.625 INFO:teuthology.orchestra.run.trial032.stderr:dumped all 2026-02-20T23:02:08.635 INFO:teuthology.orchestra.run.trial032.stdout:{"pg_ready":true,"pg_map":{"version":1577,"stamp":"2026-02-20T23:02:07.132984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":291496,"kb_used_data":12280,"kb_used_omap":8111,"kb_used_meta":271056,"kb_avail":282824024,"statfs":{"total":289910292480,"available":289611800576,"internally_reserved":0,"allocated":12574720,"data_stored":7280563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":8306057,"internal_metadata":277561975},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.000922"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:53.257842+0000","last_change":"2026-02-20T23:01:53.257842+0000","last_active":"2026-02-20T23:01:53.257842+0000","last_peered":"2026-02-20T23:01:53.257842+0000","last_clean":"2026-02-20T23:01:53.257842+0000","last_became_active":"2026-02-20T22:24:55.877789+0000","last_became_peered":"2026-02-20T22:24:55.877789+0000","last_unstale":"2026-02-20T23:01:53.257842+0000","last_undegraded":"2026-02-20T23:01:53.257842+0000","last_fullsized":"2026-02-20T23:01:53.257842+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:53.257779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:53.257779+0000","last_clean_scrub_stamp":"2026-02-20T23:01:53.257779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T01:06:00.932313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091060200000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:52.277577+0000","last_change":"2026-02-20T23:01:52.277577+0000","last_active":"2026-02-20T23:01:52.277577+0000","last_peered":"2026-02-20T23:01:52.277577+0000","last_clean":"2026-02-20T23:01:52.277577+0000","last_became_active":"2026-02-20T22:24:55.877705+0000","last_became_peered":"2026-02-20T22:24:55.877705+0000","last_unstale":"2026-02-20T23:01:52.277577+0000","last_undegraded":"2026-02-20T23:01:52.277577+0000","last_fullsized":"2026-02-20T23:01:52.277577+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:52.276968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:52.276968+0000","last_clean_scrub_stamp":"2026-02-20T23:01:52.276968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T09:52:40.696829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051339400000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:51.275726+0000","last_change":"2026-02-20T23:01:51.275726+0000","last_active":"2026-02-20T23:01:51.275726+0000","last_peered":"2026-02-20T23:01:51.275726+0000","last_clean":"2026-02-20T23:01:51.275726+0000","last_became_active":"2026-02-20T22:24:55.877840+0000","last_became_peered":"2026-02-20T22:24:55.877840+0000","last_unstale":"2026-02-20T23:01:51.275726+0000","last_undegraded":"2026-02-20T23:01:51.275726+0000","last_fullsized":"2026-02-20T23:01:51.275726+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:51.275643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:51.275643+0000","last_clean_scrub_stamp":"2026-02-20T23:01:51.275643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:39:14.727602+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.000652212,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":938,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:50.281026+0000","last_change":"2026-02-20T23:01:50.281026+0000","last_active":"2026-02-20T23:01:50.281026+0000","last_peered":"2026-02-20T23:01:50.281026+0000","last_clean":"2026-02-20T23:01:50.281026+0000","last_became_active":"2026-02-20T22:24:55.877939+0000","last_became_peered":"2026-02-20T22:24:55.877939+0000","last_unstale":"2026-02-20T23:01:50.281026+0000","last_undegraded":"2026-02-20T23:01:50.281026+0000","last_fullsized":"2026-02-20T23:01:50.281026+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:50.280174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:50.280174+0000","last_clean_scrub_stamp":"2026-02-20T23:01:50.280174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T04:36:31.806545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082465299999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"461'3","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:48.205048+0000","last_change":"2026-02-20T23:01:48.205048+0000","last_active":"2026-02-20T23:01:48.205048+0000","last_peered":"2026-02-20T23:01:48.205048+0000","last_clean":"2026-02-20T23:01:48.205048+0000","last_became_active":"2026-02-20T22:24:55.877956+0000","last_became_peered":"2026-02-20T22:24:55.877956+0000","last_unstale":"2026-02-20T23:01:48.205048+0000","last_undegraded":"2026-02-20T23:01:48.205048+0000","last_fullsized":"2026-02-20T23:01:48.205048+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'3","last_scrub_stamp":"2026-02-20T23:01:48.204984+0000","last_deep_scrub":"461'3","last_deep_scrub_stamp":"2026-02-20T23:01:48.204984+0000","last_clean_scrub_stamp":"2026-02-20T23:01:48.204984+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:31:21.478283+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00074389300000000003,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":939,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:49.949407+0000","last_change":"2026-02-20T23:01:49.949407+0000","last_active":"2026-02-20T23:01:49.949407+0000","last_peered":"2026-02-20T23:01:49.949407+0000","last_clean":"2026-02-20T23:01:49.949407+0000","last_became_active":"2026-02-20T22:24:55.877900+0000","last_became_peered":"2026-02-20T22:24:55.877900+0000","last_unstale":"2026-02-20T23:01:49.949407+0000","last_undegraded":"2026-02-20T23:01:49.949407+0000","last_fullsized":"2026-02-20T23:01:49.949407+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:49.949330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:49.949330+0000","last_clean_scrub_stamp":"2026-02-20T23:01:49.949330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T08:09:44.234623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035580099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:48.977692+0000","last_change":"2026-02-20T23:01:48.977692+0000","last_active":"2026-02-20T23:01:48.977692+0000","last_peered":"2026-02-20T23:01:48.977692+0000","last_clean":"2026-02-20T23:01:48.977692+0000","last_became_active":"2026-02-20T22:24:55.877763+0000","last_became_peered":"2026-02-20T22:24:55.877763+0000","last_unstale":"2026-02-20T23:01:48.977692+0000","last_undegraded":"2026-02-20T23:01:48.977692+0000","last_fullsized":"2026-02-20T23:01:48.977692+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2026-02-20T23:01:48.977624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2026-02-20T23:01:48.977624+0000","last_clean_scrub_stamp":"2026-02-20T23:01:48.977624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T05:15:35.146438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000319334,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"461'2","reported_seq":940,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:49.267147+0000","last_change":"2026-02-20T23:01:49.267147+0000","last_active":"2026-02-20T23:01:49.267147+0000","last_peered":"2026-02-20T23:01:49.267147+0000","last_clean":"2026-02-20T23:01:49.267147+0000","last_became_active":"2026-02-20T22:24:55.877256+0000","last_became_peered":"2026-02-20T22:24:55.877256+0000","last_unstale":"2026-02-20T23:01:49.267147+0000","last_undegraded":"2026-02-20T23:01:49.267147+0000","last_fullsized":"2026-02-20T23:01:49.267147+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'2","last_scrub_stamp":"2026-02-20T23:01:49.267075+0000","last_deep_scrub":"461'2","last_deep_scrub_stamp":"2026-02-20T23:01:49.267075+0000","last_clean_scrub_stamp":"2026-02-20T23:01:49.267075+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2026-02-21T23:02:40.344388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044139,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"461'74","reported_seq":1030,"reported_epoch":461,"state":"active+clean","last_fresh":"2026-02-20T23:01:48.304786+0000","last_change":"2026-02-20T23:01:48.304786+0000","last_active":"2026-02-20T23:01:48.304786+0000","last_peered":"2026-02-20T23:01:48.304786+0000","last_clean":"2026-02-20T23:01:48.304786+0000","last_became_active":"2026-02-20T22:24:53.869732+0000","last_became_peered":"2026-02-20T22:24:53.869732+0000","last_unstale":"2026-02-20T23:01:48.304786+0000","last_undegraded":"2026-02-20T23:01:48.304786+0000","last_fullsized":"2026-02-20T23:01:48.304786+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'74","last_scrub_stamp":"2026-02-20T23:01:48.304695+0000","last_deep_scrub":"461'74","last_deep_scrub_stamp":"2026-02-20T23:01:48.304695+0000","last_clean_scrub_stamp":"2026-02-20T23:01:48.304695+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2026-02-22T07:06:09.621188+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738818,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84744,"kb_used_data":3704,"kb_used_omap":2983,"kb_used_meta":78040,"kb_avail":94287096,"statfs":{"total":96636764160,"available":96549986304,"internally_reserved":0,"allocated":3792896,"data_stored":2033263,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3055400,"internal_metadata":79913176},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738818,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85328,"kb_used_data":4288,"kb_used_omap":1999,"kb_used_meta":79024,"kb_avail":94286512,"statfs":{"total":96636764160,"available":96549388288,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2047842,"internal_metadata":80920734},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738817,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121424,"kb_used_data":4288,"kb_used_omap":3127,"kb_used_meta":113992,"kb_avail":94250416,"statfs":{"total":96636764160,"available":96512425984,"internally_reserved":0,"allocated":4390912,"data_stored":2623650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3202815,"internal_metadata":116728065},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2026-02-20T23:02:08.637 DEBUG:teuthology.orchestra.run.trial032:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2026-02-20T23:02:08.802 INFO:teuthology.misc:Shutting down mds daemons... 2026-02-20T23:02:08.802 INFO:teuthology.misc:Shutting down osd daemons... 2026-02-20T23:02:08.802 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2026-02-20T23:02:08.802 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:02:08.900 INFO:tasks.ceph.cluster1.osd.0:Stopped 2026-02-20T23:02:08.900 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2026-02-20T23:02:08.901 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:02:09.000 INFO:tasks.ceph.cluster1.osd.1:Stopped 2026-02-20T23:02:09.001 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2026-02-20T23:02:09.001 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:02:09.084 INFO:tasks.ceph.cluster1.osd.2:Stopped 2026-02-20T23:02:09.084 INFO:teuthology.misc:Shutting down mgr daemons... 2026-02-20T23:02:09.085 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2026-02-20T23:02:09.085 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:02:09.112 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2026-02-20T23:02:09.112 INFO:teuthology.misc:Shutting down mon daemons... 2026-02-20T23:02:09.112 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2026-02-20T23:02:09.112 INFO:teuthology.orchestra.run:waiting for 300 2026-02-20T23:02:09.174 INFO:tasks.ceph.cluster1.mon.a:Stopped 2026-02-20T23:02:09.175 INFO:tasks.ceph:Checking cluster log for badness... 2026-02-20T23:02:09.175 DEBUG:teuthology.orchestra.run.trial032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2026-02-20T23:02:09.225 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T23:02:09.225 DEBUG:teuthology.orchestra.run.trial032:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2026-02-20T23:02:09.291 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T23:02:09.291 DEBUG:teuthology.orchestra.run.trial032:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2026-02-20T23:02:09.350 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@trial032.front.sepia.ceph.com 2026-02-20T23:02:09.350 DEBUG:teuthology.orchestra.run.trial032:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2026-02-20T23:02:09.409 INFO:tasks.ceph:Archiving mon data... 2026-02-20T23:02:09.410 DEBUG:teuthology.misc:Transferring archived files from trial032:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/data/cluster1.mon.a.tgz 2026-02-20T23:02:09.410 DEBUG:teuthology.orchestra.run.trial032:> mktemp 2026-02-20T23:02:09.454 INFO:teuthology.orchestra.run.trial032.stdout:/tmp/tmp.hBBxH3fGFz 2026-02-20T23:02:09.454 DEBUG:teuthology.orchestra.run.trial032:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.hBBxH3fGFz 2026-02-20T23:02:09.553 DEBUG:teuthology.orchestra.run.trial032:> sudo chmod 0666 /tmp/tmp.hBBxH3fGFz 2026-02-20T23:02:09.613 DEBUG:teuthology.orchestra.remote:trial032:/tmp/tmp.hBBxH3fGFz is 396KB 2026-02-20T23:02:09.717 DEBUG:teuthology.orchestra.run.trial032:> rm -fr /tmp/tmp.hBBxH3fGFz 2026-02-20T23:02:09.722 INFO:tasks.ceph:Cleaning ceph cluster... 2026-02-20T23:02:09.723 DEBUG:teuthology.orchestra.run.trial032:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T23:02:09.765 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2026-02-20T23:02:09.819 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T23:02:09.823 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2026-02-20T23:02:09.824 INFO:tasks.ceph:Archiving crash dumps... 2026-02-20T23:02:09.827 DEBUG:teuthology.misc:Transferring archived files from trial032:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial032/crash 2026-02-20T23:02:09.828 DEBUG:teuthology.orchestra.run.trial032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T23:02:09.871 DEBUG:teuthology.misc:Transferring archived files from trial107:/var/lib/ceph/crash to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial107/crash 2026-02-20T23:02:09.873 DEBUG:teuthology.orchestra.run.trial107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2026-02-20T23:02:09.880 INFO:tasks.ceph:Compressing logs... 2026-02-20T23:02:09.880 DEBUG:teuthology.orchestra.run.trial032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T23:02:09.916 DEBUG:teuthology.orchestra.run.trial107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19060.log 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.19060.log: /var/log/ceph/cluster1-osd.1.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19060.log.gz 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18997.log 2026-02-20T23:02:09.944 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18751.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18997.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.15949.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18997.log.gz 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18796.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18751.log.gz 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.15949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21006.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15949.log.gz 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21214.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18796.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18796.log.gz 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21006.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21006.log.gz 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21032.log 2026-02-20T23:02:09.945 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21214.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.15984.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr: 93.0% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21032.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21032.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18822.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.15984.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15984.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster2-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18981.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18822.log: gzip -5 --verbose -- /var/log/ceph/cluster1.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18822.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18981.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18981.log.gz 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21290.log 2026-02-20T23:02:09.946 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19108.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1.log: /var/log/ceph/cluster1-client.admin.21290.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21290.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19160.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5/var/log/ceph/cluster1-client.admin.19108.log: --verbose -- /var/log/ceph/cluster1-client.admin.15925.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19108.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21084.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.19160.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19160.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.15925.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15925.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.20980.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18907.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21084.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21084.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18874.log 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.20980.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.20980.log.gz 2026-02-20T23:02:09.947 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18907.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18907.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18906.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18848.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18874.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18874.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21240.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18906.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18906.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18848.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18848.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.15859.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18776.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21240.log: /var/log/ceph/cluster1-client.admin.15859.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21240.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15859.log.gz 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.15787.log 2026-02-20T23:02:09.948 INFO:teuthology.orchestra.run.trial032.stderr: 88.2% -- replaced with /var/log/ceph/cluster1.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18776.log: gzip -5 --verbose 80.4% -- -- replaced with /var/log/ceph/cluster1-client.admin.18776.log.gz /var/log/ceph/cluster1-osd.0.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr: 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21162.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.15787.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15787.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.16083.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21162.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21162.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18726.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.16083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19056.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.16083.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18726.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21110.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18726.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.19056.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12123.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19056.log.gz 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21110.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.15901.log 2026-02-20T23:02:09.949 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21110.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19134.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/ceph.tmp-client.admin.12123.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12123.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.15901.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.15901.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18982.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.19134.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19134.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5/var/log/ceph/cluster1-client.admin.18982.log: --verbose -- /var/log/ceph/cluster1-client.admin.21188.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18982.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21266.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21188.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.18908.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21188.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21266.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21058.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21266.log.gz 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.18908.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.21136.log 2026-02-20T23:02:09.950 INFO:teuthology.orchestra.run.trial032.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.18908.log.gz 2026-02-20T23:02:09.951 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21058.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21058.log.gz 2026-02-20T23:02:09.951 INFO:teuthology.orchestra.run.trial032.stderr:/var/log/ceph/cluster1-client.admin.21136.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.21136.log.gz 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53762.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44705.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53762.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53861.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53762.log.gz 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51952.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39478.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53861.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.44705.log.gz 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51952.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31971.log 2026-02-20T23:02:09.974 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53861.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51952.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39478.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48018.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43737.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31971.log: /var/log/ceph/cluster2-client.mirror.48018.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38083.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39478.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48018.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43737.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23868.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31971.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38083.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19834.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23868.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43737.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38083.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19970.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19834.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48844.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 80.4% -- replaced with /var/log/ceph/cluster1-client.admin.19834.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27761.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19970.log: /var/log/ceph/cluster2-client.mirror.48844.log: 65.1% -- replaced with /var/log/ceph/cluster1-client.mirror.19970.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32973.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48183.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42084.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48844.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48183.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47128.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48183.log.gz 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.42084.log: --verbose -- /var/log/ceph/cluster2-client.mirror.47293.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25369.log 2026-02-20T23:02:09.975 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47128.log: 87.9%/var/log/ceph/cluster2-client.mirror.47293.log: -- replaced with /var/log/ceph/cluster1-client.mirror.42084.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24193.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47128.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25369.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.23868.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52445.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27761.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24193.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47293.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53549.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46705.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.24193.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52445.log: /var/log/ceph/cluster2-client.mirror.53549.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51547.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52445.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40389.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46705.log: /var/log/ceph/cluster1-client.mirror.51547.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51547.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52770.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28225.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53549.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52770.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34035.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.25369.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28225.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40135.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40389.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34035.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54550.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46705.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34035.log.gz 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48493.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54550.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32931.log 2026-02-20T23:02:09.976 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54550.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48493.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40135.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37604.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48493.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52770.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24662.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49754.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37604.log: /var/log/ceph/cluster2-client.mirror.24662.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51862.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37604.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49754.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18741.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32931.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49754.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51862.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51862.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28116.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18741.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25703.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18741.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29124.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28225.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23827.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25703.log: /var/log/ceph/cluster2-client.mirror.29124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53629.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.25703.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29124.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19202.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23827.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24662.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53629.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23891.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.23827.log.gz 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50804.log 2026-02-20T23:02:09.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19202.log: 0.0%/var/log/ceph/cluster2-client.mirror.23891.log: -- replaced with /var/log/ceph/cluster2-client.admin.19202.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42580.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50804.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39275.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50804.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42580.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24704.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39275.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39275.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45818.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.24704.log: /var/log/ceph/cluster1-client.mirror.34413.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25816.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45818.log: /var/log/ceph/cluster1-client.mirror.34413.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42580.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28116.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45818.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.23891.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24704.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33761.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19919.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25816.log: /var/log/ceph/cluster1-client.mirror.33761.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33761.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37217.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34413.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46748.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19919.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29517.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 5.8% -- replaced with /var/log/ceph/cluster1-client.mirror.19919.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37217.log: /var/log/ceph/cluster2-client.mirror.46748.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44386.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46748.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29517.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46873.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37217.log.gz 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23187.log 2026-02-20T23:02:09.978 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45517.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39106.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46873.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45517.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.23187.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43104.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45517.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44386.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44657.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39106.log: /var/log/ceph/cluster1-client.mirror.43104.log: 87.8%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.39106.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.admin.24394.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.25816.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44657.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43104.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41490.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44657.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25530.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24394.log: /var/log/ceph/cluster2-client.mirror.41490.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41490.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.24394.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53287.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35837.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25530.log: /var/log/ceph/cluster2-client.mirror.53287.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55258.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23211.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35837.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49692.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55258.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35837.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.25530.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23211.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38059.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49692.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.23211.log.gz 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43883.log 2026-02-20T23:02:09.979 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49692.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19228.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43883.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38059.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50512.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 87.9%/var/log/ceph/cluster2-client.admin.19228.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43883.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19228.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39347.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50512.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37580.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50512.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41863.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39347.log: /var/log/ceph/cluster1-client.mirror.37580.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51119.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39347.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41863.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37580.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41863.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36277.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51119.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53652.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51119.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52882.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36277.log: /var/log/ceph/cluster1-client.mirror.53652.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53652.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43414.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip/var/log/ceph/cluster2-client.mirror.52882.log: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42924.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36277.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43414.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.52882.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33918.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23391.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42924.log: /var/log/ceph/cluster1-client.mirror.33918.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33918.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26503.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54572.log 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43414.log.gz 2026-02-20T23:02:09.980 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42105.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26503.log: /var/log/ceph/cluster2-client.mirror.54572.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42924.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32033.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42105.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40618.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32033.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32033.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34131.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.26503.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42105.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53508.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40618.log: /var/log/ceph/cluster1-client.mirror.34131.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40521.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53508.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34131.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.40618.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52087.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40521.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29049.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.53508.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40521.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52087.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52087.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42783.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29049.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42600.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42783.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54723.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42783.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42600.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54472.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31380.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54723.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54472.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.23391.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28657.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54472.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5/var/log/ceph/cluster2-client.mirror.31380.log: --verbose -- /var/log/ceph/cluster2-client.mirror.45381.log 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28657.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31380.log.gz 2026-02-20T23:02:09.981 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42600.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47727.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.28657.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29852.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47727.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55607.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23911.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55607.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36865.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55607.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23911.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45381.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31617.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29852.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36865.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.53374.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47727.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36865.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31617.log: /var/log/ceph/cluster2-client.mirror.0.53374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43518.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54513.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43518.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43518.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32250.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.23911.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24169.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54513.log: /var/log/ceph/cluster2-client.mirror.32250.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37249.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.54513.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24169.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29215.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.53374.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.24169.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37249.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32736.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29215.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47521.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.37249.log.gz 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.32736.log: /var/log/ceph/cluster1-client.mirror.34300.log 2026-02-20T23:02:09.982 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29215.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47521.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39412.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34300.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47521.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34300.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30795.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51907.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39412.log: /var/log/ceph/cluster2-client.mirror.30795.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28245.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51907.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33131.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51907.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26128.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45232.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.39412.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26128.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41815.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26128.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34716.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34107.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45232.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36464.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41815.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34107.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34716.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38865.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34107.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36464.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28245.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45746.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.38865.log: --verbose -- /var/log/ceph/cluster1-client.mirror.40847.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45746.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38865.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49423.log 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40847.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45746.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36464.log.gz 2026-02-20T23:02:09.983 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42903.log 2026-02-20T23:02:09.984 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40847.log.gz 2026-02-20T23:02:09.984 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose/var/log/ceph/cluster2-client.mirror.49423.log: -- /var/log/ceph/cluster2-client.mirror.43904.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35885.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49423.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43904.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36203.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.42903.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35885.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41020.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35885.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36203.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20158.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41020.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43904.log.gzgzip 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54407.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20158.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41020.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24063.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.20158.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36203.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54407.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49237.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54407.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48059.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24063.log: /var/log/ceph/cluster2-client.mirror.49237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36731.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49237.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23129.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.24063.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36731.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39777.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48059.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23129.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36731.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44100.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39777.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38194.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39777.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44100.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31010.log 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44100.log.gz 2026-02-20T23:02:09.990 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38194.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.23129.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24929.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31010.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38194.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42125.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31010.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24929.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55996.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42125.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23328.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55996.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55996.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47376.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45965.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.24929.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47376.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42125.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28331.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47376.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25979.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45965.log: /var/log/ceph/cluster1-client.mirror.28331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47934.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25979.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45965.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30031.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50354.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30031.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.47934.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27661.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30031.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.25979.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50354.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23307.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50354.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27661.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34945.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.23328.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23307.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41179.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.23307.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35684.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34945.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41179.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46830.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35684.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47190.log 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28331.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41179.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46830.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.35684.log.gz 2026-02-20T23:02:09.991 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27618.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37416.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27618.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40114.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47190.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37416.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36086.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27661.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40114.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46830.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23731.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40114.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36086.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37008.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36086.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54892.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23731.log: /var/log/ceph/cluster2-client.mirror.37008.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47335.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37416.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.54892.log: -- replaced with /var/log/ceph/cluster2-client.mirror.23731.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51007.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54892.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.37008.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47335.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49340.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47335.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51007.log: gzip 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51007.log.gz -5 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.52482.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49340.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46983.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52482.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53883.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27618.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52482.log.gz 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46983.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28517.log 2026-02-20T23:02:09.992 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53883.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.19690.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28517.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23435.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49340.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28517.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.3.19690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47459.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23435.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26417.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46983.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47459.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42855.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 88.4%/var/log/ceph/cluster1-client.admin.26417.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47459.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40272.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.26417.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42855.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27815.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42855.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40272.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24037.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.40272.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27815.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24010.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 92.1% -- replaced with /var/log/ceph/cluster2-client.mirror.3.19690.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24037.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44681.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.24037.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29195.log 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.24010.log.gz 2026-02-20T23:02:09.994 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44681.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27109.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29195.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44681.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38767.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27109.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37130.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23933.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.29195.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37130.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18898.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37130.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38767.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23933.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52274.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50377.log 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18898.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.23933.log.gz 2026-02-20T23:02:09.995 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27815.log.gz 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52274.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32714.log 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50377.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52274.log.gz 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50377.log.gz 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35222.log 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32714.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34966.log 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32714.log.gz 2026-02-20T23:02:09.996 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46419.log 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35222.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27109.log.gz 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32519.log 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35222.log.gz 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46419.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29379.log 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32519.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49216.log 2026-02-20T23:02:09.997 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32519.log.gz 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46419.log.gz 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29379.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34966.log.gz 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29379.log.gz 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29994.log 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50489.log 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49216.log: /var/log/ceph/cluster2-client.mirror.29994.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55727.log 2026-02-20T23:02:09.998 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29994.log.gz 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50489.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32208.log 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50489.log.gz 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55727.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55727.log.gz 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44028.log 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34011.log 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49216.log.gz 2026-02-20T23:02:09.999 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46549.log 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32208.log.gz 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44028.log.gz 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34011.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41157.log 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34011.log.gz 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46549.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47355.log 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46549.log.gz 2026-02-20T23:02:10.000 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41157.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33232.log 2026-02-20T23:02:10.001 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47355.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52919.log 2026-02-20T23:02:10.001 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.33232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53927.log 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52919.log: gzip -5 --verbose -- 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41157.log.gz 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr: /var/log/ceph/cluster2-client.mirror.1.19656.log 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52919.log.gz 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53927.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53927.log.gz 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster2-client.mirror.44494.log 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.1.19656.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47355.log.gz 2026-02-20T23:02:10.002 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55582.log 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44494.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28268.log 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44494.log.gz 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55582.log: 95.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33232.log.gz 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36327.log 0.0% 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.55582.log.gz 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28268.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25859.log 2026-02-20T23:02:10.003 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36327.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47955.log 2026-02-20T23:02:10.004 INFO:teuthology.orchestra.run.trial107.stderr: 93.1%/var/log/ceph/cluster1-client.mirror.25859.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36327.log.gz 2026-02-20T23:02:10.004 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.19656.log.gz 2026-02-20T23:02:10.004 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39600.log 2026-02-20T23:02:10.004 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47955.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22628.log 2026-02-20T23:02:10.004 INFO:teuthology.orchestra.run.trial107.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.39600.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47955.log.gz 2026-02-20T23:02:10.005 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39600.log.gz 2026-02-20T23:02:10.005 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster1-client.mirror.19953.log 2026-02-20T23:02:10.005 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22628.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40912.log 2026-02-20T23:02:10.005 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19953.log: 65.1% -- replaced with /var/log/ceph/cluster1-client.mirror.19953.log.gz 2026-02-20T23:02:10.006 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44187.log 2026-02-20T23:02:10.006 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40912.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43247.log 2026-02-20T23:02:10.006 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44187.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43345.log 2026-02-20T23:02:10.006 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43247.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28268.log.gz 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27398.log 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44187.log.gz 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43345.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40912.log.gz 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43345.log.gz 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.25859.log.gz 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31674.log 2026-02-20T23:02:10.007 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43247.log.gz 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27398.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36817.log 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31674.log: gzip -5 --verbose 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31674.log.gz 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster2-client.mirror.49051.log 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36817.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.22628.log.gz 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36572.log 2026-02-20T23:02:10.008 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36817.log.gz 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49051.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26529.log 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36572.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49051.log.gz 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39753.log 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36572.log.gz 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34897.log 2026-02-20T23:02:10.009 INFO:teuthology.orchestra.run.trial107.stderr: 89.3% -- replaced with /var/log/ceph/cluster2-client.mirror.26529.log.gz 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39753.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50017.log 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.34897.log: -- replaced with /var/log/ceph/cluster2-client.mirror.39753.log.gz 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25232.log 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34897.log.gz 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48369.log 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50017.log.gz -- replaced with /var/log/ceph/cluster2-client.mirror.27398.log.gz 2026-02-20T23:02:10.010 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25232.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43679.log 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48369.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.25232.log.gz 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45589.log 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48369.log.gz 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33697.log 2026-02-20T23:02:10.011 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45589.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51682.log 2026-02-20T23:02:10.012 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45589.log.gz 2026-02-20T23:02:10.012 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33697.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40871.log 2026-02-20T23:02:10.012 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43811.log 2026-02-20T23:02:10.012 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43679.log.gz 2026-02-20T23:02:10.012 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51682.log.gz 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35150.log 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40871.log.gz 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25349.log 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43811.log.gz 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35150.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33480.log 2026-02-20T23:02:10.013 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35150.log.gz 2026-02-20T23:02:10.014 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25349.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37854.log 2026-02-20T23:02:10.014 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33480.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38391.log 2026-02-20T23:02:10.014 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33480.log.gz 2026-02-20T23:02:10.014 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37854.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42831.log 2026-02-20T23:02:10.014 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37854.log.gz 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48638.log 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38391.log.gz/var/log/ceph/cluster1-client.mirror.42831.log: 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54108.log 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42831.log.gz 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33618.log 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48638.log.gz 2026-02-20T23:02:10.015 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54108.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56074.log 2026-02-20T23:02:10.016 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33618.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54108.log.gz 2026-02-20T23:02:10.016 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19504.log 2026-02-20T23:02:10.016 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.56074.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56074.log.gz 2026-02-20T23:02:10.016 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster2-client.mirror.54298.log 2026-02-20T23:02:10.016 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19504.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19504.log.gz 2026-02-20T23:02:10.017 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.25349.log.gz 2026-02-20T23:02:10.017 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50894.log 2026-02-20T23:02:10.017 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54298.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24526.log 2026-02-20T23:02:10.017 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54298.log.gz 2026-02-20T23:02:10.017 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50894.log: gzip -5 --verbose -- 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50894.log.gz 2026-02-20T23:02:10.018 INFO:teuthology.orchestra.run.trial107.stderr: /var/log/ceph/cluster2-client.mirror.32694.log 2026-02-20T23:02:10.018 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24526.log: gzip -5 --verbose 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.24526.log.gz 2026-02-20T23:02:10.018 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster1-client.mirror.41313.log 2026-02-20T23:02:10.018 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32694.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.31291.log 2026-02-20T23:02:10.018 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41313.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32694.log.gz 2026-02-20T23:02:10.019 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36158.log 2026-02-20T23:02:10.019 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41313.log.gz 2026-02-20T23:02:10.019 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.31291.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52526.log 2026-02-20T23:02:10.019 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36158.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43763.log 2026-02-20T23:02:10.019 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36158.log.gz 2026-02-20T23:02:10.020 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52526.log: gzip 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52526.log.gz 2026-02-20T23:02:10.020 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41265.log 2026-02-20T23:02:10.020 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43763.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54064.log 2026-02-20T23:02:10.020 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43763.log.gz 2026-02-20T23:02:10.021 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41265.log: gzip -5 --verbose 87.9% -- /var/log/ceph/cluster2-client.mirror.52250.log 2026-02-20T23:02:10.021 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41265.log.gz 2026-02-20T23:02:10.021 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54064.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54064.log.gz 2026-02-20T23:02:10.021 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26859.log 2026-02-20T23:02:10.021 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52250.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25926.log 2026-02-20T23:02:10.022 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.26859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43456.log 2026-02-20T23:02:10.022 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster1-client.admin.26859.log.gz 2026-02-20T23:02:10.022 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.25926.log: gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.25926.log.gz 2026-02-20T23:02:10.022 INFO:teuthology.orchestra.run.trial107.stderr: -- /var/log/ceph/cluster2-client.mirror.29714.log 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43456.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27044.log 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29714.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43456.log.gz 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29714.log.gz 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr: --verbose -- /var/log/ceph/cluster1-client.mirror.44052.log 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27044.log: gzip -5 --verbose -- 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27044.log.gz 2026-02-20T23:02:10.023 INFO:teuthology.orchestra.run.trial107.stderr: /var/log/ceph/cluster2-client.mirror.25392.log 2026-02-20T23:02:10.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44052.log: gzip 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44052.log.gz 2026-02-20T23:02:10.024 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41673.log 2026-02-20T23:02:10.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25392.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27208.log 2026-02-20T23:02:10.025 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49196.log 2026-02-20T23:02:10.025 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27208.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41673.log.gz 2026-02-20T23:02:10.025 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35660.log 2026-02-20T23:02:10.026 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49196.log: gzip -5 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49196.log.gz 2026-02-20T23:02:10.026 INFO:teuthology.orchestra.run.trial107.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.29815.log 90.2% 2026-02-20T23:02:10.026 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.27208.log.gz 2026-02-20T23:02:10.026 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35660.log: gzip -5 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35660.log.gz 2026-02-20T23:02:10.026 INFO:teuthology.orchestra.run.trial107.stderr: --verbose -- /var/log/ceph/cluster2-client.mirror.36486.log 2026-02-20T23:02:10.027 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31654.log 2026-02-20T23:02:10.027 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36486.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52966.log 2026-02-20T23:02:10.027 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31654.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36486.log.gzgzip 2026-02-20T23:02:10.027 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30338.log 2026-02-20T23:02:10.027 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.25392.log.gz 2026-02-20T23:02:10.028 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52966.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31654.log.gz 2026-02-20T23:02:10.028 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52966.log.gz 2026-02-20T23:02:10.028 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20021.log 2026-02-20T23:02:10.028 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30338.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29756.log 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.20021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47810.log 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr: 57.0% -- replaced with /var/log/ceph/cluster1-client.mirror.20021.log.gz 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30338.log.gz 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29756.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29756.log.gz 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22487.log 2026-02-20T23:02:10.029 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27575.log 2026-02-20T23:02:10.030 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47810.log: /var/log/ceph/cluster2-client.mirror.22487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27420.log 2026-02-20T23:02:10.030 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.22487.log.gz/var/log/ceph/cluster2-client.mirror.27575.log: 2026-02-20T23:02:10.030 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49072.log 2026-02-20T23:02:10.030 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47810.log.gz 2026-02-20T23:02:10.031 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47541.log 2026-02-20T23:02:10.031 INFO:teuthology.orchestra.run.trial107.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.27420.log.gz 2026-02-20T23:02:10.031 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41137.log 2026-02-20T23:02:10.031 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49072.log.gz 2026-02-20T23:02:10.031 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33151.log 2026-02-20T23:02:10.032 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47541.log: /var/log/ceph/cluster2-client.mirror.41137.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46063.log 2026-02-20T23:02:10.032 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33151.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33151.log.gz 2026-02-20T23:02:10.032 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41137.log.gz 2026-02-20T23:02:10.033 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47541.log.gz 2026-02-20T23:02:10.033 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27575.log.gz 2026-02-20T23:02:10.033 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24095.log 2026-02-20T23:02:10.033 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46063.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23705.log 2026-02-20T23:02:10.034 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46063.log.gz/var/log/ceph/cluster2-client.mirror.24095.log: 2026-02-20T23:02:10.034 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.46873.log 2026-02-20T23:02:10.034 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49949.log 2026-02-20T23:02:10.034 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24095.log.gz/var/log/ceph/cluster2-client.mirror.2.46873.log: 2026-02-20T23:02:10.034 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32378.log 2026-02-20T23:02:10.035 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.23705.log.gz 2026-02-20T23:02:10.035 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38538.log 2026-02-20T23:02:10.035 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.49949.log: /var/log/ceph/cluster2-client.mirror.32378.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.54171.log 2026-02-20T23:02:10.035 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38538.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49949.log.gz 2026-02-20T23:02:10.036 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32378.log.gz 2026-02-20T23:02:10.036 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55065.log 2026-02-20T23:02:10.036 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38538.log.gz 2026-02-20T23:02:10.036 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.2.54171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27881.log 2026-02-20T23:02:10.037 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55065.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28417.log 2026-02-20T23:02:10.037 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27881.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55065.log.gz 2026-02-20T23:02:10.037 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55892.log 2026-02-20T23:02:10.038 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28417.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28417.log.gz 2026-02-20T23:02:10.038 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16087.log 2026-02-20T23:02:10.039 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55892.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55892.log.gz 2026-02-20T23:02:10.039 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45866.log 2026-02-20T23:02:10.039 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.16087.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16087.log.gz 2026-02-20T23:02:10.039 INFO:teuthology.orchestra.run.trial107.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.54171.log.gz 2026-02-20T23:02:10.040 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19786.log 2026-02-20T23:02:10.040 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45866.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39323.log 2026-02-20T23:02:10.040 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.27881.log.gz 2026-02-20T23:02:10.040 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.45866.log.gz 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19786.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19786.log.gz 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31795.log 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr: 93.7%/var/log/ceph/cluster1-client.mirror.39323.log: -- replaced with /var/log/ceph/cluster2-client.mirror.2.46873.log.gz 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32598.log 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39323.log.gz 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31795.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28818.log 2026-02-20T23:02:10.041 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31795.log.gz 2026-02-20T23:02:10.042 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32598.log: gzip -5 --verbose 80.0% -- -- replaced with /var/log/ceph/cluster2-client.mirror.32598.log.gz /var/log/ceph/cluster2-client.mirror.31210.log 2026-02-20T23:02:10.042 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.042 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28818.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42187.log 2026-02-20T23:02:10.043 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31210.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20316.log 2026-02-20T23:02:10.043 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31210.log.gz 2026-02-20T23:02:10.043 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48679.log 2026-02-20T23:02:10.043 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26332.log 2026-02-20T23:02:10.043 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42187.log.gz 2026-02-20T23:02:10.044 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20316.log: /var/log/ceph/cluster2-client.mirror.48679.log: 91.2%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.28818.log.gz -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26777.log 2026-02-20T23:02:10.044 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.044 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48679.log.gz 2026-02-20T23:02:10.044 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26332.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.20316.log.gz 2026-02-20T23:02:10.044 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41884.log 2026-02-20T23:02:10.045 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26777.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24639.log 2026-02-20T23:02:10.045 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41884.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34986.log 2026-02-20T23:02:10.046 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24639.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41884.log.gz 2026-02-20T23:02:10.047 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.26332.log.gz 2026-02-20T23:02:10.047 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25727.log 2026-02-20T23:02:10.047 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34986.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50444.log 2026-02-20T23:02:10.048 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25727.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.25727.log.gz 2026-02-20T23:02:10.048 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34986.log.gz 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial032.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53570.log 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50444.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44124.log 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50444.log.gz 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53570.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53570.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52685.log 2026-02-20T23:02:10.049 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.050 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44124.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.24639.log.gz 2026-02-20T23:02:10.050 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44124.log.gz 2026-02-20T23:02:10.050 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22535.log 2026-02-20T23:02:10.050 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46525.log 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52685.log: /var/log/ceph/cluster1-client.mirror.22535.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32536.log 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52685.log.gz 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46525.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.22535.log.gz 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial032.stderr: 91.0% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial107.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51997.log 2026-02-20T23:02:10.051 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.46525.log.gz 2026-02-20T23:02:10.052 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32536.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26289.log 2026-02-20T23:02:10.052 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51997.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31227.log 2026-02-20T23:02:10.052 INFO:teuthology.orchestra.run.trial107.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51997.log.gz 2026-02-20T23:02:10.052 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42757.log 2026-02-20T23:02:10.053 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31227.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32536.log.gz 2026-02-20T23:02:10.053 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38502.log 2026-02-20T23:02:10.053 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31227.log.gz 2026-02-20T23:02:10.054 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42757.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55918.log 2026-02-20T23:02:10.054 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38502.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42757.log.gz 2026-02-20T23:02:10.054 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38502.log.gz 2026-02-20T23:02:10.054 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29087.log 2026-02-20T23:02:10.055 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55918.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50939.log 2026-02-20T23:02:10.055 INFO:teuthology.orchestra.run.trial107.stderr: 0.0%/var/log/ceph/cluster2-client.mirror.29087.log: -- replaced with /var/log/ceph/cluster2-client.admin.55918.log.gz 2026-02-20T23:02:10.055 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29087.log.gz 2026-02-20T23:02:10.055 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.26289.log.gz 2026-02-20T23:02:10.055 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41767.log 2026-02-20T23:02:10.056 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36351.log 2026-02-20T23:02:10.056 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50939.log: /var/log/ceph/cluster1-client.mirror.41767.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27352.log 2026-02-20T23:02:10.056 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50939.log.gz 2026-02-20T23:02:10.056 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36351.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41767.log.gz 2026-02-20T23:02:10.057 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35813.log 2026-02-20T23:02:10.057 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36351.log.gz 2026-02-20T23:02:10.057 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55108.log 2026-02-20T23:02:10.057 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27352.log: /var/log/ceph/cluster2-client.mirror.35813.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48782.log 2026-02-20T23:02:10.057 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35813.log.gz 2026-02-20T23:02:10.058 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55108.log: 84.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55108.log.gz 2026-02-20T23:02:10.058 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30736.log 2026-02-20T23:02:10.058 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26690.log 2026-02-20T23:02:10.059 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30736.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30736.log.gz 2026-02-20T23:02:10.059 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48782.log.gz 2026-02-20T23:02:10.060 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52642.log 2026-02-20T23:02:10.060 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29677.log 2026-02-20T23:02:10.061 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52642.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52642.log.gz 2026-02-20T23:02:10.061 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27352.log.gz 2026-02-20T23:02:10.061 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50152.log 2026-02-20T23:02:10.062 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31168.log 2026-02-20T23:02:10.062 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29677.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29677.log.gz 2026-02-20T23:02:10.062 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50152.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50152.log.gz 2026-02-20T23:02:10.062 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45929.log 2026-02-20T23:02:10.063 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31168.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31168.log.gz 2026-02-20T23:02:10.063 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.26690.log.gz 2026-02-20T23:02:10.063 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36245.log 2026-02-20T23:02:10.064 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42036.log 2026-02-20T23:02:10.064 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45929.log: /var/log/ceph/cluster2-client.mirror.36245.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37756.log 2026-02-20T23:02:10.064 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42036.log: 89.9% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42036.log.gz 2026-02-20T23:02:10.065 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36245.log.gz 2026-02-20T23:02:10.065 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45929.log.gz 2026-02-20T23:02:10.065 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51614.log 2026-02-20T23:02:10.065 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37756.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55751.log 2026-02-20T23:02:10.066 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51614.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51614.log.gz 2026-02-20T23:02:10.066 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37756.log.gzgzip 2026-02-20T23:02:10.066 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27987.log 2026-02-20T23:02:10.066 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25839.log 2026-02-20T23:02:10.066 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.55751.log: /var/log/ceph/cluster1-client.mirror.27987.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.55751.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54702.log 2026-02-20T23:02:10.067 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.069 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25839.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27987.log.gz 2026-02-20T23:02:10.070 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.25839.log.gz 2026-02-20T23:02:10.074 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48100.log 2026-02-20T23:02:10.074 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54702.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19479.log 2026-02-20T23:02:10.074 INFO:teuthology.orchestra.run.trial107.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54702.log.gz/var/log/ceph/cluster2-client.mirror.48100.log: 2026-02-20T23:02:10.074 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51749.log 2026-02-20T23:02:10.074 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53265.log 2026-02-20T23:02:10.075 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19479.log: /var/log/ceph/cluster1-client.mirror.51749.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19479.log.gz 2026-02-20T23:02:10.075 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51749.log.gz 2026-02-20T23:02:10.075 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30633.log 2026-02-20T23:02:10.075 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48100.log.gz 2026-02-20T23:02:10.075 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25208.log 2026-02-20T23:02:10.076 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53265.log: /var/log/ceph/cluster2-client.mirror.30633.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18707.log 2026-02-20T23:02:10.076 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25208.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53265.log.gz 2026-02-20T23:02:10.076 INFO:teuthology.orchestra.run.trial107.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.25208.log.gz 2026-02-20T23:02:10.076 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2026-02-20T23:02:10.077 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29694.log 2026-02-20T23:02:10.077 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18707.log: /var/log/ceph/cluster2-mon.a.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18707.log.gz 2026-02-20T23:02:10.078 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18766.log 2026-02-20T23:02:10.078 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29694.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29694.log.gz 2026-02-20T23:02:10.079 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40799.log 2026-02-20T23:02:10.079 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18766.log: 80.6% -- replaced with /var/log/ceph/cluster2-client.admin.18766.log.gz 2026-02-20T23:02:10.080 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48886.log 2026-02-20T23:02:10.080 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40799.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40799.log.gz 2026-02-20T23:02:10.081 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44428.log 2026-02-20T23:02:10.081 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48886.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48886.log.gz 2026-02-20T23:02:10.082 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39914.log 2026-02-20T23:02:10.082 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44428.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.3.31291.log.gz 2026-02-20T23:02:10.082 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44428.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44468.log 2026-02-20T23:02:10.083 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.083 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48328.log 2026-02-20T23:02:10.084 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44468.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44468.log.gz 2026-02-20T23:02:10.084 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.39914.log.gz 2026-02-20T23:02:10.084 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28159.log 2026-02-20T23:02:10.084 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54681.log 2026-02-20T23:02:10.085 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48328.log: /var/log/ceph/cluster1-client.mirror.28159.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48328.log.gz 2026-02-20T23:02:10.085 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39576.log 2026-02-20T23:02:10.086 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54681.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54681.log.gz 2026-02-20T23:02:10.086 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48927.log 2026-02-20T23:02:10.087 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39576.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39576.log.gz 2026-02-20T23:02:10.087 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31575.log 2026-02-20T23:02:10.088 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48927.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48927.log.gz 2026-02-20T23:02:10.088 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38317.log 2026-02-20T23:02:10.088 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31575.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28159.log.gz 2026-02-20T23:02:10.089 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31575.log.gz 2026-02-20T23:02:10.089 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25280.log 2026-02-20T23:02:10.089 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45887.log 2026-02-20T23:02:10.090 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38317.log: /var/log/ceph/cluster1-client.mirror.25280.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.25280.log.gz 2026-02-20T23:02:10.090 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38317.log.gz 2026-02-20T23:02:10.091 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44746.log 2026-02-20T23:02:10.091 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41839.log 2026-02-20T23:02:10.092 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45887.log: /var/log/ceph/cluster1-client.mirror.44746.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45887.log.gz 2026-02-20T23:02:10.092 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44746.log.gz 2026-02-20T23:02:10.092 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47004.log 2026-02-20T23:02:10.093 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41839.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41839.log.gz 2026-02-20T23:02:10.093 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43128.log 2026-02-20T23:02:10.094 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35909.log 2026-02-20T23:02:10.094 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47004.log: /var/log/ceph/cluster1-client.mirror.43128.log: 87.9% 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47004.log.gz 2026-02-20T23:02:10.094 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43128.log.gz 2026-02-20T23:02:10.094 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46851.log 2026-02-20T23:02:10.095 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26577.log 2026-02-20T23:02:10.095 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35909.log: /var/log/ceph/cluster2-client.mirror.46851.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35909.log.gz 2026-02-20T23:02:10.095 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46851.log.gz 2026-02-20T23:02:10.095 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28637.log 2026-02-20T23:02:10.096 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18864.log 2026-02-20T23:02:10.096 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26577.log: /var/log/ceph/cluster2-client.mirror.28637.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.26577.log.gz 2026-02-20T23:02:10.096 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.28637.log.gz 2026-02-20T23:02:10.097 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54530.log 2026-02-20T23:02:10.097 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42944.log 2026-02-20T23:02:10.097 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18864.log: /var/log/ceph/cluster2-client.mirror.54530.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18864.log.gz 2026-02-20T23:02:10.097 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39552.log 2026-02-20T23:02:10.098 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42944.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54530.log.gz 2026-02-20T23:02:10.099 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.42944.log.gz 2026-02-20T23:02:10.099 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43542.log 2026-02-20T23:02:10.099 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39936.log 2026-02-20T23:02:10.100 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39552.log: /var/log/ceph/cluster1-client.mirror.43542.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39552.log.gz 2026-02-20T23:02:10.100 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43542.log.gz 2026-02-20T23:02:10.101 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23103.log 2026-02-20T23:02:10.102 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45842.log 2026-02-20T23:02:10.102 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39936.log: /var/log/ceph/cluster1-client.admin.23103.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.23103.log.gz 2026-02-20T23:02:10.102 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39936.log.gz 2026-02-20T23:02:10.103 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20192.log 2026-02-20T23:02:10.104 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45842.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45842.log.gz 2026-02-20T23:02:10.104 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48307.log 2026-02-20T23:02:10.104 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20192.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.20192.log.gz 2026-02-20T23:02:10.105 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51232.log 2026-02-20T23:02:10.106 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48307.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48307.log.gz 2026-02-20T23:02:10.106 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29289.log 2026-02-20T23:02:10.106 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23348.log 2026-02-20T23:02:10.107 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51232.log: /var/log/ceph/cluster2-client.mirror.29289.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51232.log.gz 2026-02-20T23:02:10.107 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54935.log 2026-02-20T23:02:10.112 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23348.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.23348.log.gz 2026-02-20T23:02:10.112 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54765.log 2026-02-20T23:02:10.113 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54935.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54935.log.gz 2026-02-20T23:02:10.113 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52109.log 2026-02-20T23:02:10.114 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54765.log: 85.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54765.log.gz 2026-02-20T23:02:10.114 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40970.log 2026-02-20T23:02:10.115 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52109.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52109.log.gz 2026-02-20T23:02:10.115 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24238.log 2026-02-20T23:02:10.116 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40970.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40970.log.gz 2026-02-20T23:02:10.116 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.19532.log 2026-02-20T23:02:10.120 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24238.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.24238.log.gz 2026-02-20T23:02:10.120 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24726.log 2026-02-20T23:02:10.121 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.1.19532.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29289.log.gz 2026-02-20T23:02:10.121 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34644.log 2026-02-20T23:02:10.124 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24726.log: 94.0% -- replaced with /var/log/ceph/cluster1-client.mirror.1.19532.log.gz 2026-02-20T23:02:10.124 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38952.log 2026-02-20T23:02:10.125 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34644.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34644.log.gz 2026-02-20T23:02:10.125 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53308.log 2026-02-20T23:02:10.126 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38952.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38952.log.gz 2026-02-20T23:02:10.126 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47851.log 2026-02-20T23:02:10.127 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53308.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53308.log.gz 2026-02-20T23:02:10.127 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41514.log 2026-02-20T23:02:10.129 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47851.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47851.log.gz 2026-02-20T23:02:10.130 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42261.log 2026-02-20T23:02:10.130 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41514.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.41514.log.gz 2026-02-20T23:02:10.131 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19987.log 2026-02-20T23:02:10.131 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42261.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42261.log.gz 2026-02-20T23:02:10.132 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46451.log 2026-02-20T23:02:10.132 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19987.log: 65.3% -- replaced with /var/log/ceph/cluster1-client.mirror.19987.log.gz 2026-02-20T23:02:10.132 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53783.log 2026-02-20T23:02:10.133 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46451.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46451.log.gz 2026-02-20T23:02:10.134 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52339.log 2026-02-20T23:02:10.134 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53783.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53783.log.gz 2026-02-20T23:02:10.135 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40497.log 2026-02-20T23:02:10.135 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52339.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52339.log.gz 2026-02-20T23:02:10.136 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41562.log 2026-02-20T23:02:10.136 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40497.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40497.log.gz 2026-02-20T23:02:10.136 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37806.log 2026-02-20T23:02:10.137 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41562.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41562.log.gz 2026-02-20T23:02:10.137 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49402.log 2026-02-20T23:02:10.138 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37806.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37806.log.gz 2026-02-20T23:02:10.138 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55775.log 2026-02-20T23:02:10.140 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49402.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49402.log.gz 2026-02-20T23:02:10.140 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49883.log 2026-02-20T23:02:10.141 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55775.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55775.log.gz 2026-02-20T23:02:10.141 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49547.log 2026-02-20T23:02:10.142 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.49883.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49883.log.gz 2026-02-20T23:02:10.142 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48266.log 2026-02-20T23:02:10.143 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49547.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49547.log.gz 2026-02-20T23:02:10.143 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32852.log 2026-02-20T23:02:10.144 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48266.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48266.log.gz 2026-02-20T23:02:10.144 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51254.log 2026-02-20T23:02:10.145 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32852.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32852.log.gz 2026-02-20T23:02:10.145 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55557.log 2026-02-20T23:02:10.146 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51254.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51254.log.gz 2026-02-20T23:02:10.146 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50129.log 2026-02-20T23:02:10.147 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55557.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55557.log.gz 2026-02-20T23:02:10.147 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39665.log 2026-02-20T23:02:10.148 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50129.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50129.log.gz 2026-02-20T23:02:10.148 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45401.log 2026-02-20T23:02:10.149 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39665.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39665.log.gz 2026-02-20T23:02:10.150 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46236.log 2026-02-20T23:02:10.151 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45401.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45401.log.gz 2026-02-20T23:02:10.152 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54492.log 2026-02-20T23:02:10.152 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46236.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46236.log.gz 2026-02-20T23:02:10.152 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43787.log 2026-02-20T23:02:10.154 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54492.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54492.log.gz 2026-02-20T23:02:10.155 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41926.log 2026-02-20T23:02:10.155 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43787.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43787.log.gz 2026-02-20T23:02:10.156 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19902.log 2026-02-20T23:02:10.157 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41926.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41926.log.gz 2026-02-20T23:02:10.157 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39849.log 2026-02-20T23:02:10.157 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19902.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.19902.log.gz 2026-02-20T23:02:10.158 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56022.log 2026-02-20T23:02:10.158 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39849.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39849.log.gz 2026-02-20T23:02:10.159 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42060.log 2026-02-20T23:02:10.159 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.56022.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56022.log.gz 2026-02-20T23:02:10.159 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39994.log 2026-02-20T23:02:10.160 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42060.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42060.log.gz 2026-02-20T23:02:10.160 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50242.log 2026-02-20T23:02:10.161 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39994.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39994.log.gz 2026-02-20T23:02:10.161 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49258.log 2026-02-20T23:02:10.162 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50242.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50242.log.gz 2026-02-20T23:02:10.162 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50174.log 2026-02-20T23:02:10.163 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49258.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49258.log.gz 2026-02-20T23:02:10.163 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46152.log 2026-02-20T23:02:10.164 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50174.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50174.log.gz 2026-02-20T23:02:10.164 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40344.log 2026-02-20T23:02:10.166 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46152.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46152.log.gz 2026-02-20T23:02:10.166 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50534.log 2026-02-20T23:02:10.167 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40344.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40344.log.gz 2026-02-20T23:02:10.167 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45565.log 2026-02-20T23:02:10.168 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50534.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50534.log.gz 2026-02-20T23:02:10.168 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27306.log 2026-02-20T23:02:10.169 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45565.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45565.log.gz 2026-02-20T23:02:10.169 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47211.log 2026-02-20T23:02:10.170 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27306.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27306.log.gz 2026-02-20T23:02:10.170 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30773.log 2026-02-20T23:02:10.171 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47211.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47211.log.gz 2026-02-20T23:02:10.171 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54743.log 2026-02-20T23:02:10.172 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30773.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.30773.log.gz 2026-02-20T23:02:10.172 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53824.log 2026-02-20T23:02:10.174 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54743.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54743.log.gz 2026-02-20T23:02:10.174 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35174.log 2026-02-20T23:02:10.175 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53824.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53824.log.gz 2026-02-20T23:02:10.175 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30078.log 2026-02-20T23:02:10.176 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35174.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35174.log.gz 2026-02-20T23:02:10.176 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50039.log 2026-02-20T23:02:10.186 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.2.30078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48596.log 2026-02-20T23:02:10.186 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50039.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50039.log.gz 2026-02-20T23:02:10.189 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32973.log.gz 2026-02-20T23:02:10.189 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45360.log 2026-02-20T23:02:10.191 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48596.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48596.log.gz 2026-02-20T23:02:10.191 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48348.log 2026-02-20T23:02:10.192 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45360.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45360.log.gz 2026-02-20T23:02:10.192 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29613.log 2026-02-20T23:02:10.194 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48348.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48348.log.gz 2026-02-20T23:02:10.194 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32756.log 2026-02-20T23:02:10.195 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29613.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29613.log.gz 2026-02-20T23:02:10.195 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42351.log 2026-02-20T23:02:10.196 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32756.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32756.log.gz 2026-02-20T23:02:10.196 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56132.log 2026-02-20T23:02:10.197 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42351.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.42351.log.gz 2026-02-20T23:02:10.198 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.25552.log 2026-02-20T23:02:10.198 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.56132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56132.log.gz 2026-02-20T23:02:10.199 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26444.log 2026-02-20T23:02:10.201 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.0.25552.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.25552.log.gz 2026-02-20T23:02:10.202 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46597.log 2026-02-20T23:02:10.202 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.26444.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.26444.log.gz 2026-02-20T23:02:10.203 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33358.log 2026-02-20T23:02:10.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46597.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46597.log.gz 2026-02-20T23:02:10.204 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41092.log 2026-02-20T23:02:10.204 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33358.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33358.log.gz 2026-02-20T23:02:10.205 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19810.log 2026-02-20T23:02:10.205 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41092.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41092.log.gz 2026-02-20T23:02:10.206 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48865.log 2026-02-20T23:02:10.206 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19810.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19810.log.gz 2026-02-20T23:02:10.206 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38172.log 2026-02-20T23:02:10.207 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48865.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48865.log.gz 2026-02-20T23:02:10.207 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31417.log 2026-02-20T23:02:10.209 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38172.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38172.log.gz 2026-02-20T23:02:10.209 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41215.log 2026-02-20T23:02:10.210 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31417.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31417.log.gz 2026-02-20T23:02:10.210 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35996.log 2026-02-20T23:02:10.211 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41215.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41215.log.gz 2026-02-20T23:02:10.211 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30495.log 2026-02-20T23:02:10.212 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35996.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35996.log.gz 2026-02-20T23:02:10.212 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33417.log 2026-02-20T23:02:10.213 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30495.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30495.log.gz 2026-02-20T23:02:10.213 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41241.log 2026-02-20T23:02:10.214 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33417.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33417.log.gz 2026-02-20T23:02:10.214 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40681.log 2026-02-20T23:02:10.215 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41241.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41241.log.gz 2026-02-20T23:02:10.215 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28354.log 2026-02-20T23:02:10.216 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40681.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40681.log.gz 2026-02-20T23:02:10.217 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53487.log 2026-02-20T23:02:10.221 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28354.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28354.log.gz 2026-02-20T23:02:10.221 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43152.log 2026-02-20T23:02:10.221 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53487.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53487.log.gz 2026-02-20T23:02:10.222 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33941.log 2026-02-20T23:02:10.222 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43152.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43152.log.gz 2026-02-20T23:02:10.223 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38564.log 2026-02-20T23:02:10.224 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33941.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33941.log.gz 2026-02-20T23:02:10.224 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23413.log 2026-02-20T23:02:10.224 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38564.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38564.log.gz 2026-02-20T23:02:10.225 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38439.log 2026-02-20T23:02:10.226 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23413.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.23413.log.gz 2026-02-20T23:02:10.226 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43056.log 2026-02-20T23:02:10.227 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38439.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38439.log.gz 2026-02-20T23:02:10.227 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43193.log 2026-02-20T23:02:10.227 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43056.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43056.log.gz 2026-02-20T23:02:10.228 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35612.log 2026-02-20T23:02:10.229 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43193.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.43193.log.gz 2026-02-20T23:02:10.234 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33559.log 2026-02-20T23:02:10.234 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35612.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35612.log.gz 2026-02-20T23:02:10.234 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41791.log 2026-02-20T23:02:10.235 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33559.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33559.log.gz 2026-02-20T23:02:10.235 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27132.log 2026-02-20T23:02:10.236 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41791.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41791.log.gz 2026-02-20T23:02:10.236 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52208.log 2026-02-20T23:02:10.240 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27132.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27132.log.gz 2026-02-20T23:02:10.240 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30184.log 2026-02-20T23:02:10.241 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52208.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52208.log.gz 2026-02-20T23:02:10.241 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39010.log 2026-02-20T23:02:10.242 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30184.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30184.log.gz 2026-02-20T23:02:10.242 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46768.log 2026-02-20T23:02:10.243 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39010.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39010.log.gz 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.55258.log.gz 2026-02-20T23:02:10.243 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.243 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40996.log 2026-02-20T23:02:10.244 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46768.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.32736.log.gz 2026-02-20T23:02:10.245 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31954.log 2026-02-20T23:02:10.245 INFO:teuthology.orchestra.run.trial107.stderr: 90.6%/var/log/ceph/cluster2-client.mirror.40996.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46768.log.gz 2026-02-20T23:02:10.246 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41464.log 2026-02-20T23:02:10.246 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40996.log.gz 2026-02-20T23:02:10.247 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24619.log 2026-02-20T23:02:10.247 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31954.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31954.log.gz 2026-02-20T23:02:10.247 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34348.log 2026-02-20T23:02:10.247 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41116.log 2026-02-20T23:02:10.247 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24619.log: /var/log/ceph/cluster1-client.mirror.34348.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41464.log.gz 2026-02-20T23:02:10.248 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34348.log.gz 2026-02-20T23:02:10.248 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32129.log 2026-02-20T23:02:10.248 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41116.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49382.log 2026-02-20T23:02:10.248 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32129.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41116.log.gz 2026-02-20T23:02:10.248 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33089.log 2026-02-20T23:02:10.249 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32129.log.gz 2026-02-20T23:02:10.249 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49382.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45304.log 2026-02-20T23:02:10.249 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33089.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49382.log.gz 2026-02-20T23:02:10.249 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40892.log 2026-02-20T23:02:10.250 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33089.log.gz 2026-02-20T23:02:10.250 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45304.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45304.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45794.log 2026-02-20T23:02:10.250 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.250 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40892.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.24619.log.gz 2026-02-20T23:02:10.251 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40892.log.gz 2026-02-20T23:02:10.251 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48245.log 2026-02-20T23:02:10.252 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48700.log 2026-02-20T23:02:10.252 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.48245.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45794.log.gz 2026-02-20T23:02:10.252 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31148.log 2026-02-20T23:02:10.253 INFO:teuthology.orchestra.run.trial107.stderr: 88.2%/var/log/ceph/cluster2-client.mirror.48700.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48245.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38664.log 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31148.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52404.log 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48700.log.gz 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38664.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31052.log 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31148.log.gz 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38664.log.gz 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52404.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.52404.log.gz 2026-02-20T23:02:10.254 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49444.log 2026-02-20T23:02:10.255 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23755.log 2026-02-20T23:02:10.255 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31052.log: /var/log/ceph/cluster2-client.mirror.49444.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31052.log.gz 2026-02-20T23:02:10.255 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46356.log 2026-02-20T23:02:10.255 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49444.log.gz 2026-02-20T23:02:10.255 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46664.log 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.23755.log.gz 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46356.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48080.log 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46664.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.46356.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38685.log 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.46664.log.gz 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48080.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48080.log.gz 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31032.log 2026-02-20T23:02:10.256 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.55108.log 2026-02-20T23:02:10.258 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38685.log: /var/log/ceph/cluster2-client.mirror.31032.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38685.log.gz 2026-02-20T23:02:10.258 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48576.log 2026-02-20T23:02:10.258 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37531.log 2026-02-20T23:02:10.259 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.1.55108.log: /var/log/ceph/cluster2-client.mirror.48576.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48576.log.gz 2026-02-20T23:02:10.259 INFO:teuthology.orchestra.run.trial107.stderr: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.55108.log.gz 2026-02-20T23:02:10.260 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48038.log 2026-02-20T23:02:10.260 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50197.log 2026-02-20T23:02:10.260 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37531.log: /var/log/ceph/cluster2-client.mirror.48038.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37531.log.gz 2026-02-20T23:02:10.261 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44768.log 2026-02-20T23:02:10.261 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50197.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50197.log.gz 2026-02-20T23:02:10.262 INFO:teuthology.orchestra.run.trial107.stderr: 90.6%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55236.log 2026-02-20T23:02:10.262 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48038.log.gz 2026-02-20T23:02:10.262 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44768.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44768.log.gz 2026-02-20T23:02:10.263 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52019.log 2026-02-20T23:02:10.263 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.22973.log 2026-02-20T23:02:10.263 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55236.log: /var/log/ceph/cluster1-client.mirror.52019.log: 87.9% 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52019.log.gz 2026-02-20T23:02:10.263 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55236.log.gz 2026-02-20T23:02:10.264 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38727.log 2026-02-20T23:02:10.264 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47624.log 2026-02-20T23:02:10.265 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.3.22973.log: /var/log/ceph/cluster2-client.mirror.38727.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38727.log.gz 2026-02-20T23:02:10.266 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.3.22973.log.gz 2026-02-20T23:02:10.267 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50332.log 2026-02-20T23:02:10.267 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39130.log 2026-02-20T23:02:10.267 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47624.log: /var/log/ceph/cluster1-client.mirror.50332.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50332.log.gz 2026-02-20T23:02:10.267 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47624.log.gz 2026-02-20T23:02:10.268 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29635.log 2026-02-20T23:02:10.268 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48555.log 2026-02-20T23:02:10.268 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39130.log: /var/log/ceph/cluster2-client.mirror.29635.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39130.log.gz 2026-02-20T23:02:10.269 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28557.log 2026-02-20T23:02:10.269 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48555.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48555.log.gz 2026-02-20T23:02:10.270 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36620.log 2026-02-20T23:02:10.270 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28557.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.28557.log.gz 2026-02-20T23:02:10.271 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26001.log 2026-02-20T23:02:10.271 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36620.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36620.log.gz 2026-02-20T23:02:10.272 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29475.log 2026-02-20T23:02:10.272 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26001.log: 77.9% -- replaced with /var/log/ceph/cluster2-client.mirror.26001.log.gz 2026-02-20T23:02:10.272 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40431.log 2026-02-20T23:02:10.273 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29475.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29475.log.gz 2026-02-20T23:02:10.273 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44317.log 2026-02-20T23:02:10.274 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40431.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40431.log.gz 2026-02-20T23:02:10.275 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36709.log 2026-02-20T23:02:10.275 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44317.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44317.log.gz 2026-02-20T23:02:10.276 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24876.log 2026-02-20T23:02:10.277 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36709.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36709.log.gz 2026-02-20T23:02:10.277 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26312.log 2026-02-20T23:02:10.278 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24876.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.24876.log.gz 2026-02-20T23:02:10.278 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19936.log 2026-02-20T23:02:10.282 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26312.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.26312.log.gz 2026-02-20T23:02:10.282 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35954.log 2026-02-20T23:02:10.282 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19936.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.19936.log.gz 2026-02-20T23:02:10.283 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49904.log 2026-02-20T23:02:10.284 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35954.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35954.log.gz 2026-02-20T23:02:10.285 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46963.log 2026-02-20T23:02:10.285 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.49904.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49904.log.gz 2026-02-20T23:02:10.285 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41586.log 2026-02-20T23:02:10.286 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46963.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46963.log.gz 2026-02-20T23:02:10.286 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53883.log.gz 2026-02-20T23:02:10.287 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36767.log 2026-02-20T23:02:10.287 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41586.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51322.log 2026-02-20T23:02:10.287 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41586.log.gz 2026-02-20T23:02:10.288 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36767.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52064.log 2026-02-20T23:02:10.288 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51322.log: 89.9% 87.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36767.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.51322.log.gz 2026-02-20T23:02:10.288 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.289 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33675.log 2026-02-20T23:02:10.289 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52064.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31131.log 2026-02-20T23:02:10.289 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52064.log.gz 2026-02-20T23:02:10.290 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33675.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31069.log 2026-02-20T23:02:10.290 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31131.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33675.log.gz 2026-02-20T23:02:10.290 INFO:teuthology.orchestra.run.trial107.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31131.log.gz 2026-02-20T23:02:10.290 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53228.log 2026-02-20T23:02:10.291 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42622.log 2026-02-20T23:02:10.291 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31069.log: /var/log/ceph/cluster2-client.mirror.53228.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53228.log.gz 2026-02-20T23:02:10.291 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30973.log 2026-02-20T23:02:10.291 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31069.log.gz 2026-02-20T23:02:10.292 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42622.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19454.log 2026-02-20T23:02:10.292 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42622.log.gz 2026-02-20T23:02:10.292 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30973.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30973.log.gz 2026-02-20T23:02:10.292 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44365.log 2026-02-20T23:02:10.293 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19454.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19454.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53054.log 2026-02-20T23:02:10.293 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.293 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50669.log 2026-02-20T23:02:10.294 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53054.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44365.log.gz 2026-02-20T23:02:10.294 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.53054.log.gz 2026-02-20T23:02:10.294 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48142.log 2026-02-20T23:02:10.294 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33168.log 2026-02-20T23:02:10.295 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50669.log: /var/log/ceph/cluster2-client.mirror.48142.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50669.log.gz 2026-02-20T23:02:10.295 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48142.log.gz 2026-02-20T23:02:10.295 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29951.log 2026-02-20T23:02:10.296 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44219.log 2026-02-20T23:02:10.296 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33168.log: /var/log/ceph/cluster2-client.mirror.29951.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.29951.log.gz 2026-02-20T23:02:10.296 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33168.log.gz 2026-02-20T23:02:10.296 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42807.log 2026-02-20T23:02:10.297 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44219.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26885.log 2026-02-20T23:02:10.297 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42807.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.44219.log.gz 2026-02-20T23:02:10.297 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42807.log.gz 2026-02-20T23:02:10.298 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50557.log 2026-02-20T23:02:10.299 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26885.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.26885.log.gz 2026-02-20T23:02:10.299 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37195.log 2026-02-20T23:02:10.299 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19124.log 2026-02-20T23:02:10.300 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50557.log: /var/log/ceph/cluster1-client.mirror.37195.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50557.log.gz 2026-02-20T23:02:10.300 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47748.log 2026-02-20T23:02:10.301 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19124.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19124.log.gz 2026-02-20T23:02:10.301 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37195.log.gz 2026-02-20T23:02:10.301 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33030.log 2026-02-20T23:02:10.301 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36399.log 2026-02-20T23:02:10.302 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47748.log: /var/log/ceph/cluster2-client.mirror.33030.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47748.log.gz 2026-02-20T23:02:10.302 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33030.log.gz 2026-02-20T23:02:10.302 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37175.log 2026-02-20T23:02:10.303 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34692.log 2026-02-20T23:02:10.303 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36399.log: /var/log/ceph/cluster2-client.mirror.37175.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36399.log.gz 2026-02-20T23:02:10.303 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22692.log 2026-02-20T23:02:10.304 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34692.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37175.log.gz 2026-02-20T23:02:10.304 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34692.log.gz 2026-02-20T23:02:10.304 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19132.log 2026-02-20T23:02:10.305 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22692.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.19531.log 2026-02-20T23:02:10.306 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19132.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19132.log.gz 2026-02-20T23:02:10.318 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47231.log 2026-02-20T23:02:10.333 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.0.19531.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30795.log.gz 2026-02-20T23:02:10.334 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45907.log 2026-02-20T23:02:10.335 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47231.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47231.log.gz 2026-02-20T23:02:10.336 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41962.log 2026-02-20T23:02:10.337 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45907.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45907.log.gz 2026-02-20T23:02:10.337 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25552.log 2026-02-20T23:02:10.338 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41962.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.41962.log.gz 2026-02-20T23:02:10.339 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37458.log 2026-02-20T23:02:10.339 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25552.log: 78.3% -- replaced with /var/log/ceph/cluster1-client.mirror.25552.log.gz 2026-02-20T23:02:10.339 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52899.log 2026-02-20T23:02:10.340 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37458.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37458.log.gz 2026-02-20T23:02:10.340 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33400.log 2026-02-20T23:02:10.341 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52899.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52899.log.gz 2026-02-20T23:02:10.342 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44838.log 2026-02-20T23:02:10.342 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33400.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33400.log.gz 2026-02-20T23:02:10.342 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51772.log 2026-02-20T23:02:10.343 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44838.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44838.log.gz 2026-02-20T23:02:10.343 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30078.log.gz 2026-02-20T23:02:10.343 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25953.log 2026-02-20T23:02:10.343 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51772.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37154.log 2026-02-20T23:02:10.344 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51772.log.gz 2026-02-20T23:02:10.344 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.25953.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30670.log 2026-02-20T23:02:10.344 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.25953.log.gz 2026-02-20T23:02:10.344 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37154.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18838.log 2026-02-20T23:02:10.345 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37154.log.gz 2026-02-20T23:02:10.345 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30670.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.27420.log 2026-02-20T23:02:10.346 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18838.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30670.log.gz 2026-02-20T23:02:10.346 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18838.log.gz 2026-02-20T23:02:10.346 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51884.log 2026-02-20T23:02:10.346 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.0.27420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45720.log 2026-02-20T23:02:10.347 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51884.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51884.log.gz 2026-02-20T23:02:10.347 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49733.log 2026-02-20T23:02:10.347 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33697.log.gz 2026-02-20T23:02:10.348 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45720.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45720.log.gz 2026-02-20T23:02:10.349 INFO:teuthology.orchestra.run.trial107.stderr: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.27420.log.gz 2026-02-20T23:02:10.350 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33010.log 2026-02-20T23:02:10.350 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49733.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16028.log 2026-02-20T23:02:10.350 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33010.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49733.log.gz 2026-02-20T23:02:10.350 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35008.log 2026-02-20T23:02:10.351 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.16028.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19354.log 2026-02-20T23:02:10.351 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16028.log.gz 2026-02-20T23:02:10.351 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33010.log.gz 2026-02-20T23:02:10.351 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35008.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55279.log 2026-02-20T23:02:10.352 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46810.log 2026-02-20T23:02:10.352 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19354.log.gz 2026-02-20T23:02:10.352 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55279.log.gz 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.35008.log.gz 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.31617.log.gz 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44293.log 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28537.log 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46810.log.gz 2026-02-20T23:02:10.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44293.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32578.log 2026-02-20T23:02:10.354 INFO:teuthology.orchestra.run.trial107.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.28537.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51569.log 2026-02-20T23:02:10.354 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44293.log.gz 2026-02-20T23:02:10.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32578.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48906.log 2026-02-20T23:02:10.354 INFO:teuthology.orchestra.run.trial107.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28537.log.gz 2026-02-20T23:02:10.354 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38367.log 2026-02-20T23:02:10.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51569.log: 87.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42439.log 2026-02-20T23:02:10.355 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51569.log.gz/var/log/ceph/cluster2-client.mirror.48906.log: 2026-02-20T23:02:10.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38367.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38367.log.gz 2026-02-20T23:02:10.356 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53245.log 2026-02-20T23:02:10.356 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47314.log 2026-02-20T23:02:10.356 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48906.log.gz 2026-02-20T23:02:10.356 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42439.log.gz 2026-02-20T23:02:10.357 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53245.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24753.log 2026-02-20T23:02:10.357 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54087.log 2026-02-20T23:02:10.357 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47314.log: /var/log/ceph/cluster1-client.mirror.24753.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53245.log.gz 2026-02-20T23:02:10.357 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47314.log.gz 2026-02-20T23:02:10.358 INFO:teuthology.orchestra.run.trial107.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.24753.log.gz 2026-02-20T23:02:10.358 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50624.log 2026-02-20T23:02:10.358 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39058.log 2026-02-20T23:02:10.358 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54087.log: /var/log/ceph/cluster1-client.mirror.50624.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54087.log.gz 2026-02-20T23:02:10.358 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50624.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47149.log 2026-02-20T23:02:10.359 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.359 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39082.log 2026-02-20T23:02:10.359 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39058.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37673.log 2026-02-20T23:02:10.359 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47149.log: /var/log/ceph/cluster1-client.mirror.39082.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39058.log.gz 2026-02-20T23:02:10.360 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39082.log.gz 2026-02-20T23:02:10.360 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47149.log.gz 2026-02-20T23:02:10.360 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28202.log 2026-02-20T23:02:10.360 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37673.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47025.log 2026-02-20T23:02:10.361 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28202.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40471.log 2026-02-20T23:02:10.361 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47025.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37673.log.gz 2026-02-20T23:02:10.361 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47025.log.gz 2026-02-20T23:02:10.362 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55409.log 2026-02-20T23:02:10.362 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36375.log 2026-02-20T23:02:10.363 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40471.log: /var/log/ceph/cluster2-client.mirror.55409.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55409.log.gz 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40471.log.gz 2026-02-20T23:02:10.363 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.363 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44341.log 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36375.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28202.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31832.log 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36375.log.gz 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41610.log 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18975.log 2026-02-20T23:02:10.364 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44341.log.gz 2026-02-20T23:02:10.365 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31832.log: /var/log/ceph/cluster1-client.mirror.41610.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31832.log.gz 2026-02-20T23:02:10.365 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55970.log 2026-02-20T23:02:10.365 INFO:teuthology.orchestra.run.trial107.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41610.log.gz 2026-02-20T23:02:10.365 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18975.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49972.log 2026-02-20T23:02:10.365 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.18975.log.gz 2026-02-20T23:02:10.366 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29931.log 2026-02-20T23:02:10.366 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55970.log: /var/log/ceph/cluster1-client.mirror.49972.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31854.log 2026-02-20T23:02:10.366 INFO:teuthology.orchestra.run.trial107.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49972.log.gz 2026-02-20T23:02:10.366 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.55970.log.gz 2026-02-20T23:02:10.366 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29931.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35198.log 2026-02-20T23:02:10.367 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31854.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29931.log.gz 2026-02-20T23:02:10.367 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54572.log.gz 2026-02-20T23:02:10.368 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24345.log 2026-02-20T23:02:10.368 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35198.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35341.log 2026-02-20T23:02:10.368 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24345.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35198.log.gz 2026-02-20T23:02:10.382 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28053.log 2026-02-20T23:02:10.382 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35341.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36954.log 2026-02-20T23:02:10.383 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28053.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35341.log.gz 2026-02-20T23:02:10.383 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25304.log 2026-02-20T23:02:10.384 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36954.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.0.19531.log.gz 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36954.log.gz 2026-02-20T23:02:10.384 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.384 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20064.log 2026-02-20T23:02:10.385 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39392.log 2026-02-20T23:02:10.385 INFO:teuthology.orchestra.run.trial107.stderr: 93.6%/var/log/ceph/cluster2-client.admin.20064.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.25304.log.gz 2026-02-20T23:02:10.385 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.28053.log.gz 2026-02-20T23:02:10.385 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20064.log.gz 2026-02-20T23:02:10.385 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30574.log 2026-02-20T23:02:10.386 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27020.log 2026-02-20T23:02:10.386 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30574.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27258.log 2026-02-20T23:02:10.386 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30574.log.gz 2026-02-20T23:02:10.387 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27020.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20299.log 2026-02-20T23:02:10.387 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27020.log.gz 2026-02-20T23:02:10.387 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39392.log.gz 2026-02-20T23:02:10.387 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27258.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.42645.log 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20299.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33782.log 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.20299.log.gz 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr: 89.5%/var/log/ceph/cluster1-client.mirror.1.42645.log: -- replaced with /var/log/ceph/cluster1-client.mirror.27258.log.gz 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40177.log 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.33782.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46260.log 2026-02-20T23:02:10.388 INFO:teuthology.orchestra.run.trial107.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33782.log.gz 2026-02-20T23:02:10.389 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31558.log 2026-02-20T23:02:10.389 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40177.log: /var/log/ceph/cluster1-client.mirror.46260.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46260.log.gz 2026-02-20T23:02:10.390 INFO:teuthology.orchestra.run.trial107.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45328.log 2026-02-20T23:02:10.390 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40177.log.gz 2026-02-20T23:02:10.390 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31558.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31558.log.gz 2026-02-20T23:02:10.390 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44076.log 2026-02-20T23:02:10.390 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28717.log 2026-02-20T23:02:10.391 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45328.log: /var/log/ceph/cluster1-client.mirror.44076.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45328.log.gz 2026-02-20T23:02:10.391 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44076.log.gz 2026-02-20T23:02:10.391 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46377.log 2026-02-20T23:02:10.392 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35267.log 2026-02-20T23:02:10.392 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28717.log.gz/var/log/ceph/cluster2-client.mirror.46377.log: 2026-02-20T23:02:10.393 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53529.log 2026-02-20T23:02:10.393 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35267.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46377.log.gz 2026-02-20T23:02:10.394 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34815.log 2026-02-20T23:02:10.394 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35267.log.gz 2026-02-20T23:02:10.394 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53529.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53529.log.gz 2026-02-20T23:02:10.398 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38705.log 2026-02-20T23:02:10.398 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34815.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32773.log 2026-02-20T23:02:10.398 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38705.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34815.log.gz 2026-02-20T23:02:10.399 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20124.log 2026-02-20T23:02:10.400 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32773.log: 89.8% 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.32773.log.gz 2026-02-20T23:02:10.400 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38705.log.gz 2026-02-20T23:02:10.400 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30053.log 2026-02-20T23:02:10.400 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.20124.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29773.log 2026-02-20T23:02:10.400 INFO:teuthology.orchestra.run.trial107.stderr: 72.0% -- replaced with /var/log/ceph/cluster2-client.admin.20124.log.gz 2026-02-20T23:02:10.401 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27705.log 2026-02-20T23:02:10.401 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30053.log: /var/log/ceph/cluster2-client.mirror.29773.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29773.log.gz 2026-02-20T23:02:10.402 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52989.log 2026-02-20T23:02:10.413 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27705.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43924.log 2026-02-20T23:02:10.414 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52989.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52989.log.gz 2026-02-20T23:02:10.425 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.1.42645.log.gz 2026-02-20T23:02:10.426 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28863.log 2026-02-20T23:02:10.427 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43924.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.43924.log.gz 2026-02-20T23:02:10.427 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19329.log 2026-02-20T23:02:10.428 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28863.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.28863.log.gz 2026-02-20T23:02:10.428 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28990.log 2026-02-20T23:02:10.428 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19329.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19329.log.gz 2026-02-20T23:02:10.429 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44886.log 2026-02-20T23:02:10.429 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28990.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.28990.log.gz 2026-02-20T23:02:10.429 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39171.log 2026-02-20T23:02:10.430 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44886.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44886.log.gz 2026-02-20T23:02:10.430 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25434.log 2026-02-20T23:02:10.431 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39171.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.39171.log.gz 2026-02-20T23:02:10.432 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30931.log 2026-02-20T23:02:10.433 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25434.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.25434.log.gz 2026-02-20T23:02:10.433 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24574.log 2026-02-20T23:02:10.434 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30931.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30931.log.gz 2026-02-20T23:02:10.434 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25256.log 2026-02-20T23:02:10.435 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24574.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.24574.log.gz 2026-02-20T23:02:10.435 INFO:teuthology.orchestra.run.trial107.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.27705.log.gz 2026-02-20T23:02:10.435 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37395.log 2026-02-20T23:02:10.436 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31469.log 2026-02-20T23:02:10.436 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.25256.log.gz/var/log/ceph/cluster1-client.mirror.37395.log: 2026-02-20T23:02:10.436 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37395.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24217.log 2026-02-20T23:02:10.436 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.437 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.31469.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40368.log 2026-02-20T23:02:10.437 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24217.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.24217.log.gz 2026-02-20T23:02:10.438 INFO:teuthology.orchestra.run.trial107.stderr: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31469.log.gz 2026-02-20T23:02:10.439 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32191.log 2026-02-20T23:02:10.440 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40368.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41988.log 2026-02-20T23:02:10.440 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32191.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.40368.log.gz 2026-02-20T23:02:10.440 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.32191.log.gz 2026-02-20T23:02:10.440 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37275.log 2026-02-20T23:02:10.441 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41988.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23529.log 2026-02-20T23:02:10.441 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41988.log.gz/var/log/ceph/cluster2-client.mirror.37275.log: 2026-02-20T23:02:10.441 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37275.log.gz 2026-02-20T23:02:10.441 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.52792.log 2026-02-20T23:02:10.442 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.23529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24598.log 2026-02-20T23:02:10.442 INFO:teuthology.orchestra.run.trial107.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.0.52792.log: -- replaced with /var/log/ceph/cluster1-client.admin.23529.log.gz 2026-02-20T23:02:10.442 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36913.log 2026-02-20T23:02:10.442 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24598.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.24598.log.gz 2026-02-20T23:02:10.443 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37965.log 2026-02-20T23:02:10.443 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.52792.log.gz 2026-02-20T23:02:10.443 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36913.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.36913.log.gz 2026-02-20T23:02:10.443 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32499.log 2026-02-20T23:02:10.444 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37965.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51817.log 2026-02-20T23:02:10.444 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32499.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37965.log.gz 2026-02-20T23:02:10.444 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42310.log 2026-02-20T23:02:10.445 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51817.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51817.log.gz 2026-02-20T23:02:10.445 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30753.log 2026-02-20T23:02:10.446 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42310.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42310.log.gz 2026-02-20T23:02:10.446 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53904.log 2026-02-20T23:02:10.446 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30753.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30753.log.gz 2026-02-20T23:02:10.447 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40221.log 2026-02-20T23:02:10.447 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53904.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53904.log.gz 2026-02-20T23:02:10.448 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16004.log 2026-02-20T23:02:10.448 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40221.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40221.log.gz 2026-02-20T23:02:10.449 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42285.log 2026-02-20T23:02:10.449 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.16004.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16004.log.gz 2026-02-20T23:02:10.449 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23076.log 2026-02-20T23:02:10.450 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42285.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42285.log.gz 2026-02-20T23:02:10.450 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30874.log 2026-02-20T23:02:10.451 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.23076.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.23076.log.gz 2026-02-20T23:02:10.451 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28497.log 2026-02-20T23:02:10.462 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30874.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28182.log 2026-02-20T23:02:10.462 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28497.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28497.log.gz 2026-02-20T23:02:10.470 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49506.log 2026-02-20T23:02:10.473 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28182.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28182.log.gz 2026-02-20T23:02:10.478 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31089.log 2026-02-20T23:02:10.478 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49506.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49506.log.gz 2026-02-20T23:02:10.481 INFO:teuthology.orchestra.run.trial032.stderr: 90.4% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2026-02-20T23:02:10.486 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45634.log 2026-02-20T23:02:10.486 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31089.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31089.log.gz 2026-02-20T23:02:10.506 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28288.log 2026-02-20T23:02:10.507 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45634.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45634.log.gz 2026-02-20T23:02:10.517 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30633.log.gz 2026-02-20T23:02:10.517 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2026-02-20T23:02:10.521 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28288.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28288.log.gz 2026-02-20T23:02:10.521 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53607.log 2026-02-20T23:02:10.525 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-osd.2.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29517.log.gz 2026-02-20T23:02:10.525 INFO:teuthology.orchestra.run.trial107.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29815.log.gz 2026-02-20T23:02:10.525 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31247.log 2026-02-20T23:02:10.525 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53607.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.19667.log 2026-02-20T23:02:10.526 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31247.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53607.log.gz 2026-02-20T23:02:10.526 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31247.log.gz 2026-02-20T23:02:10.526 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22648.log 2026-02-20T23:02:10.526 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.2.19667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26248.log 2026-02-20T23:02:10.530 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.22648.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.22648.log.gz 2026-02-20T23:02:10.531 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46087.log 2026-02-20T23:02:10.531 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26248.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.26248.log.gz 2026-02-20T23:02:10.532 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52383.log 2026-02-20T23:02:10.532 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46087.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46087.log.gz 2026-02-20T23:02:10.533 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48824.log 2026-02-20T23:02:10.533 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52383.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52383.log.gz 2026-02-20T23:02:10.534 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30226.log 2026-02-20T23:02:10.534 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48824.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48824.log.gz 2026-02-20T23:02:10.535 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34083.log 2026-02-20T23:02:10.542 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2026-02-20T23:02:10.554 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34083.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34083.log.gz 2026-02-20T23:02:10.557 INFO:teuthology.orchestra.run.trial107.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.19667.log.gz 2026-02-20T23:02:10.557 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27532.log 2026-02-20T23:02:10.570 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31812.log 2026-02-20T23:02:10.573 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27532.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27532.log.gz 2026-02-20T23:02:10.594 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54261.log 2026-02-20T23:02:10.595 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31812.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31812.log.gz 2026-02-20T23:02:10.600 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29635.log.gz 2026-02-20T23:02:10.602 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19629.log 2026-02-20T23:02:10.603 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54261.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54261.log.gz 2026-02-20T23:02:10.603 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53352.log 2026-02-20T23:02:10.604 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19629.log: 90.6% -- replaced with /var/log/ceph/cluster1-client.mirror.19629.log.gz 2026-02-20T23:02:10.604 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48121.log 2026-02-20T23:02:10.605 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53352.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53352.log.gz 2026-02-20T23:02:10.605 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27681.log 2026-02-20T23:02:10.605 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48121.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48121.log.gz 2026-02-20T23:02:10.606 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.21001.log 2026-02-20T23:02:10.609 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27681.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27681.log.gz 2026-02-20T23:02:10.610 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50962.log 2026-02-20T23:02:10.618 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.21001.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51299.log 2026-02-20T23:02:10.619 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50962.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50962.log.gz 2026-02-20T23:02:10.634 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36841.log 2026-02-20T23:02:10.634 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51299.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51299.log.gz 2026-02-20T23:02:10.637 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53287.log.gz 2026-02-20T23:02:10.638 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46332.log 2026-02-20T23:02:10.638 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36841.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.36841.log.gz 2026-02-20T23:02:10.639 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55814.log 2026-02-20T23:02:10.639 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46332.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46332.log.gz 2026-02-20T23:02:10.640 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27858.log 2026-02-20T23:02:10.640 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55814.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55814.log.gz 2026-02-20T23:02:10.640 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29835.log 2026-02-20T23:02:10.644 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27858.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27858.log.gz 2026-02-20T23:02:10.644 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40823.log 2026-02-20T23:02:10.644 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29835.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29835.log.gz 2026-02-20T23:02:10.645 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50467.log 2026-02-20T23:02:10.645 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40823.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40823.log.gz 2026-02-20T23:02:10.646 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39528.log 2026-02-20T23:02:10.646 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50467.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50467.log.gz 2026-02-20T23:02:10.646 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43638.log 2026-02-20T23:02:10.647 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39528.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.39528.log.gz 2026-02-20T23:02:10.647 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32070.log 2026-02-20T23:02:10.648 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43638.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43638.log.gz 2026-02-20T23:02:10.648 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39645.log 2026-02-20T23:02:10.649 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32070.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32070.log.gz 2026-02-20T23:02:10.649 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29894.log 2026-02-20T23:02:10.651 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39645.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39645.log.gz 2026-02-20T23:02:10.651 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42213.log 2026-02-20T23:02:10.656 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29894.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52250.log.gz 2026-02-20T23:02:10.657 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48472.log 2026-02-20T23:02:10.657 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42213.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42213.log.gz 2026-02-20T23:02:10.658 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33188.log 2026-02-20T23:02:10.659 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48472.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48472.log.gz 2026-02-20T23:02:10.660 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36303.log 2026-02-20T23:02:10.660 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33188.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33188.log.gz 2026-02-20T23:02:10.661 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40545.log 2026-02-20T23:02:10.661 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36303.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36303.log.gz 2026-02-20T23:02:10.662 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54342.log 2026-02-20T23:02:10.662 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40545.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40545.log.gz 2026-02-20T23:02:10.663 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49568.log 2026-02-20T23:02:10.663 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54342.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.54342.log.gz 2026-02-20T23:02:10.663 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45676.log 2026-02-20T23:02:10.664 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49568.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49568.log.gz 2026-02-20T23:02:10.664 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31912.log 2026-02-20T23:02:10.665 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45676.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45676.log.gz 2026-02-20T23:02:10.665 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19885.log 2026-02-20T23:02:10.666 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31912.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31912.log.gz 2026-02-20T23:02:10.666 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52228.log 2026-02-20T23:02:10.666 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19885.log: 63.5% -- replaced with /var/log/ceph/cluster1-client.mirror.19885.log.gz 2026-02-20T23:02:10.667 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22559.log 2026-02-20T23:02:10.667 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52228.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52228.log.gz 2026-02-20T23:02:10.668 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52042.log 2026-02-20T23:02:10.668 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.22559.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.22559.log.gz 2026-02-20T23:02:10.669 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50714.log 2026-02-20T23:02:10.669 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52042.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.52042.log.gz 2026-02-20T23:02:10.669 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49278.log 2026-02-20T23:02:10.670 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50714.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50714.log.gz 2026-02-20T23:02:10.670 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33539.log 2026-02-20T23:02:10.671 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49278.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49278.log.gz 2026-02-20T23:02:10.672 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27552.log 2026-02-20T23:02:10.676 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33539.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.53629.log.gz 2026-02-20T23:02:10.676 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.45102.log 2026-02-20T23:02:10.680 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27552.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27552.log.gz 2026-02-20T23:02:10.680 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26102.log 2026-02-20T23:02:10.693 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.45102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19404.log 2026-02-20T23:02:10.694 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26102.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.26102.log.gz 2026-02-20T23:02:10.697 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32250.log.gz 2026-02-20T23:02:10.698 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25328.log 2026-02-20T23:02:10.698 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19404.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19404.log.gz 2026-02-20T23:02:10.699 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47087.log 2026-02-20T23:02:10.699 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25328.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.25328.log.gz 2026-02-20T23:02:10.699 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40659.log 2026-02-20T23:02:10.700 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47087.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47087.log.gz 2026-02-20T23:02:10.700 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52173.log 2026-02-20T23:02:10.701 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40659.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.45102.log.gz 2026-02-20T23:02:10.701 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27152.log 2026-02-20T23:02:10.702 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52173.log: 43.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52173.log.gz 2026-02-20T23:02:10.702 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40659.log.gz 2026-02-20T23:02:10.702 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25796.log 2026-02-20T23:02:10.702 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37347.log 2026-02-20T23:02:10.705 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25796.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27152.log.gz 2026-02-20T23:02:10.705 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.25796.log.gz 2026-02-20T23:02:10.705 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32171.log 2026-02-20T23:02:10.706 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37347.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37347.log.gz 2026-02-20T23:02:10.706 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46618.log 2026-02-20T23:02:10.707 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32171.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45991.log 2026-02-20T23:02:10.708 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46618.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46618.log.gz 2026-02-20T23:02:10.708 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41337.log 2026-02-20T23:02:10.708 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45991.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45991.log.gz 2026-02-20T23:02:10.709 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23848.log 2026-02-20T23:02:10.710 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41337.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41337.log.gz 2026-02-20T23:02:10.710 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42966.log 2026-02-20T23:02:10.713 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23848.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.23848.log.gz 2026-02-20T23:02:10.714 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36223.log 2026-02-20T23:02:10.714 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42966.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42966.log.gz 2026-02-20T23:02:10.715 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51637.log 2026-02-20T23:02:10.716 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36223.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36223.log.gz 2026-02-20T23:02:10.717 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47707.log 2026-02-20T23:02:10.717 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51637.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51637.log.gz 2026-02-20T23:02:10.717 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55216.log 2026-02-20T23:02:10.718 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47707.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47707.log.gz 2026-02-20T23:02:10.718 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32149.log 2026-02-20T23:02:10.719 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55216.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55216.log.gz 2026-02-20T23:02:10.719 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28798.log 2026-02-20T23:02:10.720 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32149.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32149.log.gz 2026-02-20T23:02:10.720 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43321.log 2026-02-20T23:02:10.721 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28798.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.28798.log.gz 2026-02-20T23:02:10.721 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49092.log 2026-02-20T23:02:10.722 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43321.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43321.log.gz 2026-02-20T23:02:10.722 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33618.log.gz 2026-02-20T23:02:10.722 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23235.log 2026-02-20T23:02:10.723 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49092.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55344.log 2026-02-20T23:02:10.723 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23235.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.23235.log.gz 2026-02-20T23:02:10.724 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29438.log 2026-02-20T23:02:10.724 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55344.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55344.log.gz 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49092.log.gz 2026-02-20T23:02:10.724 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.724 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.33804.log 2026-02-20T23:02:10.724 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26755.log 2026-02-20T23:02:10.728 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29438.log: /var/log/ceph/cluster2-client.mirror.2.33804.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23435.log.gz 2026-02-20T23:02:10.730 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19254.log 2026-02-20T23:02:10.730 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24345.log.gz 2026-02-20T23:02:10.730 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26755.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28457.log 2026-02-20T23:02:10.731 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.26755.log.gz 2026-02-20T23:02:10.731 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19254.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19254.log.gz 2026-02-20T23:02:10.731 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34737.log 2026-02-20T23:02:10.732 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28457.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20260.log 2026-02-20T23:02:10.732 INFO:teuthology.orchestra.run.trial107.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28457.log.gz 2026-02-20T23:02:10.733 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.32272.log 2026-02-20T23:02:10.733 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20260.log: 56.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20260.log.gz 2026-02-20T23:02:10.733 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51209.log 2026-02-20T23:02:10.733 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.1.32272.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34737.log.gz 2026-02-20T23:02:10.734 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40248.log 2026-02-20T23:02:10.734 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51209.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51209.log.gz 2026-02-20T23:02:10.734 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.53138.log 2026-02-20T23:02:10.735 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40248.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40248.log.gz 2026-02-20T23:02:10.735 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24478.log 2026-02-20T23:02:10.741 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.53138.log: 91.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.53138.log.gz 2026-02-20T23:02:10.741 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40775.log 2026-02-20T23:02:10.742 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24478.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.24478.log.gz 2026-02-20T23:02:10.742 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31291.log 2026-02-20T23:02:10.743 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40775.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.40775.log.gz 2026-02-20T23:02:10.743 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32815.log 2026-02-20T23:02:10.746 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31291.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.2.33804.log.gz 2026-02-20T23:02:10.750 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46039.log 2026-02-20T23:02:10.752 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32815.log: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.1.32272.log.gz 2026-02-20T23:02:10.752 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46111.log 2026-02-20T23:02:10.753 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46039.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46039.log.gz 2026-02-20T23:02:10.753 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36934.log 2026-02-20T23:02:10.754 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46111.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.46111.log.gz 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29049.log.gz 2026-02-20T23:02:10.754 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:10.755 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27509.log 2026-02-20T23:02:10.756 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36934.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36934.log.gz 2026-02-20T23:02:10.756 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29914.log 2026-02-20T23:02:10.759 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27509.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27509.log.gz 2026-02-20T23:02:10.761 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23259.log 2026-02-20T23:02:10.761 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29914.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30554.log 2026-02-20T23:02:10.761 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29914.log.gz 2026-02-20T23:02:10.761 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23259.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.23259.log.gz 2026-02-20T23:02:10.763 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26777.log.gz 2026-02-20T23:02:10.774 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28397.log 2026-02-20T23:02:10.775 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31854.log.gz 2026-02-20T23:02:10.775 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42331.log 2026-02-20T23:02:10.775 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30554.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44269.log 2026-02-20T23:02:10.775 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28397.log: /var/log/ceph/cluster2-client.mirror.42331.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19053.log 2026-02-20T23:02:10.775 INFO:teuthology.orchestra.run.trial107.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.28397.log.gz 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44269.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31111.log 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44269.log.gz 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19053.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19053.log.gz 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42331.log.gz 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55085.log 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19379.log 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31111.log: /var/log/ceph/cluster2-client.mirror.55085.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53718.log 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19379.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19379.log.gz 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45467.log 2026-02-20T23:02:10.778 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53718.log: 86.9% 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55085.log.gz 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.53718.log.gz 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36110.log 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45467.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43369.log 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36110.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31291.log.gz 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45467.log.gz 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36110.log.gz 2026-02-20T23:02:10.779 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24145.log 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43369.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51187.log 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52424.log 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43369.log.gz 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.24145.log.gz 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18812.log 2026-02-20T23:02:10.780 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51187.log.gz 2026-02-20T23:02:10.781 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52424.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50107.log 2026-02-20T23:02:10.781 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34873.log 2026-02-20T23:02:10.782 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18812.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18812.log.gz 2026-02-20T23:02:10.782 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50107.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41538.log 2026-02-20T23:02:10.782 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50107.log.gz 2026-02-20T23:02:10.782 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34873.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34873.log.gz 2026-02-20T23:02:10.783 INFO:teuthology.orchestra.run.trial107.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52424.log.gz 2026-02-20T23:02:10.783 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53374.log 2026-02-20T23:02:10.783 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32420.log 2026-02-20T23:02:10.784 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41538.log: /var/log/ceph/cluster1-client.mirror.53374.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45206.log 2026-02-20T23:02:10.784 INFO:teuthology.orchestra.run.trial107.stderr: 84.1% -- replaced with /var/log/ceph/cluster1-client.mirror.53374.log.gz 2026-02-20T23:02:10.784 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32420.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.41538.log.gz 2026-02-20T23:02:10.784 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48390.log 2026-02-20T23:02:10.784 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45206.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30990.log 2026-02-20T23:02:10.785 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48390.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.45206.log.gz 2026-02-20T23:02:10.785 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48390.log.gz 2026-02-20T23:02:10.785 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49588.log 2026-02-20T23:02:10.786 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30990.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.26001.log 2026-02-20T23:02:10.786 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49588.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30990.log.gz 2026-02-20T23:02:10.787 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39193.log 2026-02-20T23:02:10.787 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.1.26001.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49588.log.gz 2026-02-20T23:02:10.789 INFO:teuthology.orchestra.run.trial107.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.1.26001.log.gz 2026-02-20T23:02:10.789 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33052.log 2026-02-20T23:02:10.790 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39193.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27595.log 2026-02-20T23:02:10.790 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33052.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39193.log.gz 2026-02-20T23:02:10.791 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51029.log 2026-02-20T23:02:10.794 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27595.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27595.log.gz 2026-02-20T23:02:10.795 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44726.log 2026-02-20T23:02:10.795 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51029.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51029.log.gz 2026-02-20T23:02:10.795 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26376.log 2026-02-20T23:02:10.796 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.24726.log.gz 2026-02-20T23:02:10.796 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44726.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19046.log 2026-02-20T23:02:10.797 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26376.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44726.log.gz 2026-02-20T23:02:10.797 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56108.log 2026-02-20T23:02:10.798 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19046.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19046.log.gz 2026-02-20T23:02:10.798 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26176.log 2026-02-20T23:02:10.799 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.56108.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56108.log.gz 2026-02-20T23:02:10.799 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38415.log 2026-02-20T23:02:10.800 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26176.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.26176.log.gz 2026-02-20T23:02:10.800 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29973.log 2026-02-20T23:02:10.800 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38415.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.38415.log.gz 2026-02-20T23:02:10.801 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26601.log 2026-02-20T23:02:10.822 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29973.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47045.log 2026-02-20T23:02:10.823 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26601.log: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.26601.log.gz 2026-02-20T23:02:10.840 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33131.log.gz 2026-02-20T23:02:10.840 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22670.log 2026-02-20T23:02:10.842 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47045.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47045.log.gz 2026-02-20T23:02:10.842 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34252.log 2026-02-20T23:02:10.843 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22670.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.22670.log.gz 2026-02-20T23:02:10.843 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51929.log 2026-02-20T23:02:10.844 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34252.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34252.log.gz 2026-02-20T23:02:10.844 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27282.log 2026-02-20T23:02:10.844 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51929.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51929.log.gz 2026-02-20T23:02:10.845 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38217.log 2026-02-20T23:02:10.845 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27282.log: 89.7% -- replaced with /var/log/ceph/cluster1-client.mirror.27282.log.gz 2026-02-20T23:02:10.846 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43173.log 2026-02-20T23:02:10.851 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.1.38217.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30053.log.gz 2026-02-20T23:02:10.858 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37034.log 2026-02-20T23:02:10.859 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43173.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43173.log.gz 2026-02-20T23:02:10.859 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55460.log 2026-02-20T23:02:10.860 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37034.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37034.log.gz 2026-02-20T23:02:10.860 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45423.log 2026-02-20T23:02:10.861 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55460.log: 78.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55460.log.gz 2026-02-20T23:02:10.861 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.31469.log 2026-02-20T23:02:10.862 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45423.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.45423.log.gz 2026-02-20T23:02:10.862 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49485.log 2026-02-20T23:02:10.871 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.3.31469.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38217.log.gz 2026-02-20T23:02:10.874 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47603.log 2026-02-20T23:02:10.875 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49485.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49485.log.gz 2026-02-20T23:02:10.875 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54658.log 2026-02-20T23:02:10.876 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47603.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47603.log.gz 2026-02-20T23:02:10.877 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48224.log 2026-02-20T23:02:10.878 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54658.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54658.log.gz 2026-02-20T23:02:10.878 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47583.log 2026-02-20T23:02:10.879 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48224.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48224.log.gz 2026-02-20T23:02:10.880 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50917.log 2026-02-20T23:02:10.881 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47583.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47583.log.gz 2026-02-20T23:02:10.881 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54129.log 2026-02-20T23:02:10.882 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50917.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50917.log.gz 2026-02-20T23:02:10.882 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51479.log 2026-02-20T23:02:10.883 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54129.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54129.log.gz 2026-02-20T23:02:10.883 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55429.log 2026-02-20T23:02:10.884 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51479.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51479.log.gz 2026-02-20T23:02:10.884 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39504.log 2026-02-20T23:02:10.885 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55429.log: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55429.log.gz 2026-02-20T23:02:10.886 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50084.log 2026-02-20T23:02:10.886 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39504.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39504.log.gz 2026-02-20T23:02:10.888 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35415.log 2026-02-20T23:02:10.888 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50084.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50084.log.gz 2026-02-20T23:02:10.888 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31991.log 2026-02-20T23:02:10.888 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35415.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35415.log.gz 2026-02-20T23:02:10.889 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46789.log 2026-02-20T23:02:10.889 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31991.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31991.log.gz 2026-02-20T23:02:10.890 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31595.log 2026-02-20T23:02:10.890 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46789.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46789.log.gz 2026-02-20T23:02:10.891 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19868.log 2026-02-20T23:02:10.892 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31595.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31595.log.gz 2026-02-20T23:02:10.892 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49795.log 2026-02-20T23:02:10.892 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19868.log: 71.8% -- replaced with /var/log/ceph/cluster1-client.admin.19868.log.gz 2026-02-20T23:02:10.893 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.55460.log 2026-02-20T23:02:10.893 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49795.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49795.log.gz 2026-02-20T23:02:10.893 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43566.log 2026-02-20T23:02:10.894 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.0.55460.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.55460.log.gz 2026-02-20T23:02:10.895 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55388.log 2026-02-20T23:02:10.895 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43566.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43566.log.gz 2026-02-20T23:02:10.896 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53741.log 2026-02-20T23:02:10.896 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55388.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55388.log.gz 2026-02-20T23:02:10.897 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.24753.log 2026-02-20T23:02:10.897 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.53741.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53741.log.gz 2026-02-20T23:02:10.897 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46685.log 2026-02-20T23:02:10.910 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.3.24753.log: 92.8%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39727.log 2026-02-20T23:02:10.911 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46685.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46685.log.gz 2026-02-20T23:02:10.912 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.3.24753.log.gz 2026-02-20T23:02:10.912 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33719.log 2026-02-20T23:02:10.912 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39727.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39727.log.gz 2026-02-20T23:02:10.913 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38009.log 2026-02-20T23:02:10.914 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33719.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33719.log.gz 2026-02-20T23:02:10.914 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35787.log 2026-02-20T23:02:10.914 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38009.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38009.log.gz 2026-02-20T23:02:10.915 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49836.log 2026-02-20T23:02:10.916 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35787.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35787.log.gz 2026-02-20T23:02:10.916 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22583.log 2026-02-20T23:02:10.918 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49836.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49836.log.gz 2026-02-20T23:02:10.919 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30815.log 2026-02-20T23:02:10.919 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.22583.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.22583.log.gz 2026-02-20T23:02:10.920 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41068.log 2026-02-20T23:02:10.920 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30815.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30815.log.gz 2026-02-20T23:02:10.920 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44633.log 2026-02-20T23:02:10.921 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41068.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41068.log.gz 2026-02-20T23:02:10.921 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.19531.log 2026-02-20T23:02:10.922 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44633.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44633.log.gz 2026-02-20T23:02:10.922 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43297.log 2026-02-20T23:02:10.928 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.19531.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32578.log.gz 2026-02-20T23:02:10.929 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19667.log 2026-02-20T23:02:10.929 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43297.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43297.log.gz 2026-02-20T23:02:10.930 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33517.log 2026-02-20T23:02:10.931 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.19667.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.19667.log.gz 2026-02-20T23:02:10.932 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45102.log 2026-02-20T23:02:10.932 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33517.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33517.log.gz 2026-02-20T23:02:10.933 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26470.log 2026-02-20T23:02:10.933 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45102.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.3.31469.log.gz 2026-02-20T23:02:10.933 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35463.log 2026-02-20T23:02:10.934 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26470.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.26470.log.gz 2026-02-20T23:02:10.935 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2026-02-20T23:02:10.935 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35463.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35463.log.gz 2026-02-20T23:02:10.936 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29458.log 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2.log: 87.4% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.45102.log.gz 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2.log.gz 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28010.log 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29458.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51434.log 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr: 80.2% -- replaced with /var/log/ceph/cluster2-client.mirror.29458.log.gz 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54451.log 2026-02-20T23:02:10.941 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51434.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51434.log.gz 2026-02-20T23:02:10.943 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34779.log 2026-02-20T23:02:10.943 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54451.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.54451.log.gz 2026-02-20T23:02:10.943 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47417.log 2026-02-20T23:02:10.944 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34779.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34779.log.gz 2026-02-20T23:02:10.944 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55044.log 2026-02-20T23:02:10.944 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47417.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28010.log.gz 2026-02-20T23:02:10.947 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31397.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55044.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47417.log.gz 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.55044.log.gz 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48989.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31397.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25903.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48989.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31397.log.gz 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48989.log.gz 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39825.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25903.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30653.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39825.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39825.log.gz 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40569.log 2026-02-20T23:02:10.951 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30653.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30653.log.gz 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28777.log 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40569.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40569.log.gz 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39371.log 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28777.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.28777.log.gz 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18786.log 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39371.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.39371.log.gz 2026-02-20T23:02:10.952 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.34555.log 2026-02-20T23:02:10.953 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18786.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18786.log.gz 2026-02-20T23:02:10.953 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26269.log 2026-02-20T23:02:10.974 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.0.34555.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.29289.log 2026-02-20T23:02:10.977 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26269.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.26269.log.gz 2026-02-20T23:02:10.982 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31032.log.gz 2026-02-20T23:02:10.986 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44165.log 2026-02-20T23:02:11.002 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.0.29289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53948.log 2026-02-20T23:02:11.003 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44165.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44165.log.gz 2026-02-20T23:02:11.010 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29438.log.gz 2026-02-20T23:02:11.011 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32793.log 2026-02-20T23:02:11.011 INFO:teuthology.orchestra.run.trial107.stderr: 90.3% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2026-02-20T23:02:11.011 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53948.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39687.log 2026-02-20T23:02:11.012 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32793.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32793.log.gz 2026-02-20T23:02:11.012 INFO:teuthology.orchestra.run.trial107.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53948.log.gz 2026-02-20T23:02:11.012 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36036.log 2026-02-20T23:02:11.012 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47769.log 2026-02-20T23:02:11.012 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39687.log: /var/log/ceph/cluster2-client.mirror.36036.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39687.log.gz 2026-02-20T23:02:11.013 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36036.log.gz 2026-02-20T23:02:11.013 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28374.log 2026-02-20T23:02:11.013 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42559.log 2026-02-20T23:02:11.013 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47769.log: /var/log/ceph/cluster1-client.mirror.28374.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47769.log.gz 2026-02-20T23:02:11.014 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49630.log 2026-02-20T23:02:11.014 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42559.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.42559.log.gz 2026-02-20T23:02:11.015 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35725.log 2026-02-20T23:02:11.015 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49630.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49630.log.gz 2026-02-20T23:02:11.015 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.20004.log 2026-02-20T23:02:11.017 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35725.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28374.log.gz 2026-02-20T23:02:11.017 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35725.log.gz 2026-02-20T23:02:11.017 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32440.log 2026-02-20T23:02:11.018 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44958.log 2026-02-20T23:02:11.018 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.20004.log: /var/log/ceph/cluster2-client.mirror.32440.log: 56.3% -- replaced with /var/log/ceph/cluster1-client.mirror.20004.log.gz 2026-02-20T23:02:11.018 INFO:teuthology.orchestra.run.trial107.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32440.log.gz 2026-02-20T23:02:11.018 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.53974.log 2026-02-20T23:02:11.019 INFO:teuthology.orchestra.run.trial107.stderr: 94.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45256.log 2026-02-20T23:02:11.019 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.19531.log.gz 2026-02-20T23:02:11.019 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44958.log: /var/log/ceph/cluster1-client.mirror.2.53974.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44958.log.gz 2026-02-20T23:02:11.020 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52750.log 2026-02-20T23:02:11.020 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50422.log 2026-02-20T23:02:11.020 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45256.log.gz 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr: 93.3%/var/log/ceph/cluster2-client.mirror.52750.log: -- replaced with /var/log/ceph/cluster1-client.mirror.0.29289.log.gz 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49774.log 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52750.log.gz 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37498.log 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50422.log.gz 2026-02-20T23:02:11.021 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49774.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30204.log 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50399.log 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37498.log: /var/log/ceph/cluster2-client.mirror.30204.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30204.log.gz 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.2.53974.log.gz 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.37498.log.gz 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49774.log.gz 2026-02-20T23:02:11.022 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37715.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50399.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46015.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37715.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27638.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50399.log.gz 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30532.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37715.log.gz 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46015.log.gz 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27638.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29237.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35861.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30532.log: /var/log/ceph/cluster2-client.mirror.29237.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29104.log 2026-02-20T23:02:11.024 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30532.log.gz 2026-02-20T23:02:11.025 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35861.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54278.log 2026-02-20T23:02:11.025 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35861.log.gz 2026-02-20T23:02:11.025 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29104.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48514.log 2026-02-20T23:02:11.025 INFO:teuthology.orchestra.run.trial107.stderr: 89.6%/var/log/ceph/cluster2-client.mirror.54278.log: -- replaced with /var/log/ceph/cluster2-client.mirror.29104.log.gz 2026-02-20T23:02:11.026 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30894.log 2026-02-20T23:02:11.026 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48514.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54278.log.gz 2026-02-20T23:02:11.027 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48514.log.gz 2026-02-20T23:02:11.027 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54593.log 2026-02-20T23:02:11.027 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27638.log.gz 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50309.log 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30894.log.gz/var/log/ceph/cluster1-client.mirror.54593.log: 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53974.log 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50309.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54593.log.gz 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22283.log 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50309.log.gz 2026-02-20T23:02:11.028 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53974.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50287.log 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr: 89.4%/var/log/ceph/cluster1-client.admin.22283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34529.log 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53974.log.gz 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50287.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.22283.log.gz 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38460.log 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50287.log.gz 2026-02-20T23:02:11.029 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34529.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24550.log 2026-02-20T23:02:11.030 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38460.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34529.log.gz 2026-02-20T23:02:11.030 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46501.log 2026-02-20T23:02:11.031 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24550.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38460.log.gz 2026-02-20T23:02:11.031 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.24550.log.gz 2026-02-20T23:02:11.031 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.55703.log 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46284.log 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46501.log: /var/log/ceph/cluster1-client.admin.55703.log: 0.0% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46501.log.gz 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.55703.log.gz 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33497.log 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19851.log 2026-02-20T23:02:11.032 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46284.log: /var/log/ceph/cluster2-client.mirror.33497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22336.log 2026-02-20T23:02:11.033 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46284.log.gz 2026-02-20T23:02:11.033 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19851.log: 80.5% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33497.log.gz 2026-02-20T23:02:11.033 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.19851.log.gz 2026-02-20T23:02:11.033 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32112.log 2026-02-20T23:02:11.034 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45541.log 2026-02-20T23:02:11.034 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22336.log: /var/log/ceph/cluster2-client.mirror.32112.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43273.log 2026-02-20T23:02:11.034 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32112.log.gz 2026-02-20T23:02:11.034 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45541.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51367.log 2026-02-20T23:02:11.035 INFO:teuthology.orchestra.run.trial107.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.43273.log: -- replaced with /var/log/ceph/cluster1-client.mirror.45541.log.gz 2026-02-20T23:02:11.035 INFO:teuthology.orchestra.run.trial107.stderr: 91.9% -- replaced with /var/log/ceph/cluster2-client.mirror.22336.log.gz 2026-02-20T23:02:11.035 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43273.log.gz 2026-02-20T23:02:11.035 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.23556.log 2026-02-20T23:02:11.035 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43946.log 2026-02-20T23:02:11.036 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51367.log: /var/log/ceph/cluster1-client.admin.23556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47397.log 2026-02-20T23:02:11.036 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51367.log.gz 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.23556.log.gz 2026-02-20T23:02:11.036 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.036 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43946.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48204.log 2026-02-20T23:02:11.036 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47397.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43946.log.gz 2026-02-20T23:02:11.037 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47397.log.gz 2026-02-20T23:02:11.037 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50737.log 2026-02-20T23:02:11.037 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48204.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34393.log 2026-02-20T23:02:11.037 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48204.log.gz 2026-02-20T23:02:11.038 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29736.log 2026-02-20T23:02:11.038 INFO:teuthology.orchestra.run.trial107.stderr: 87.0%/var/log/ceph/cluster2-client.mirror.34393.log: -- replaced with /var/log/ceph/cluster1-client.mirror.50737.log.gz 2026-02-20T23:02:11.038 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54320.log 2026-02-20T23:02:11.038 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29736.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44812.log 2026-02-20T23:02:11.039 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54320.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34393.log.gz 2026-02-20T23:02:11.039 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53075.log 2026-02-20T23:02:11.040 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44812.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44812.log.gz 2026-02-20T23:02:11.045 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32894.log 2026-02-20T23:02:11.046 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53075.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.53075.log.gz 2026-02-20T23:02:11.046 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24502.log 2026-02-20T23:02:11.050 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32894.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.34555.log.gz 2026-02-20T23:02:11.054 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20038.log 2026-02-20T23:02:11.054 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24502.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.24502.log.gz 2026-02-20T23:02:11.055 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35391.log 2026-02-20T23:02:11.055 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.20038.log: 94.1% 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.20038.log.gz 2026-02-20T23:02:11.055 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.30554.log.gz 2026-02-20T23:02:11.055 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35974.log 2026-02-20T23:02:11.056 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33459.log 2026-02-20T23:02:11.056 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.35974.log: -- replaced with /var/log/ceph/cluster1-client.mirror.35391.log.gz 2026-02-20T23:02:11.056 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29416.log 2026-02-20T23:02:11.057 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33459.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35974.log.gz 2026-02-20T23:02:11.057 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.15911.log 2026-02-20T23:02:11.058 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29416.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29416.log.gz 2026-02-20T23:02:11.066 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.22360.log 2026-02-20T23:02:11.066 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.15911.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.15911.log.gz 2026-02-20T23:02:11.066 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51389.log 2026-02-20T23:02:11.082 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.22360.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29973.log.gz 2026-02-20T23:02:11.083 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54149.log 2026-02-20T23:02:11.083 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51389.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47273.log 2026-02-20T23:02:11.083 INFO:teuthology.orchestra.run.trial107.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51389.log.gz 2026-02-20T23:02:11.084 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54149.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54149.log.gz 2026-02-20T23:02:11.085 INFO:teuthology.orchestra.run.trial107.stderr: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43859.log 2026-02-20T23:02:11.085 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33539.log.gz 2026-02-20T23:02:11.086 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47273.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36522.log 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47273.log.gz 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43859.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38152.log 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43859.log.gz 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36522.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24281.log 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38152.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36522.log.gz 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.3.22360.log.gz 2026-02-20T23:02:11.087 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.089 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32013.log 2026-02-20T23:02:11.089 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24281.log: 92.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38152.log.gz 2026-02-20T23:02:11.089 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.54765.log 2026-02-20T23:02:11.091 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32013.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.24281.log.gz 2026-02-20T23:02:11.091 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28905.log 2026-02-20T23:02:11.093 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.54765.log: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.3.54765.log.gz 2026-02-20T23:02:11.093 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49712.log 2026-02-20T23:02:11.094 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28905.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28905.log.gz 2026-02-20T23:02:11.095 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35124.log 2026-02-20T23:02:11.096 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49712.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49712.log.gz 2026-02-20T23:02:11.097 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.23604.log 2026-02-20T23:02:11.098 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35124.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35124.log.gz 2026-02-20T23:02:11.098 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32361.log 2026-02-20T23:02:11.106 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.1.23604.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29146.log 2026-02-20T23:02:11.107 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32361.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32361.log.gz 2026-02-20T23:02:11.107 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34757.log 2026-02-20T23:02:11.116 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29146.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.23604.log.gz 2026-02-20T23:02:11.117 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55944.log 2026-02-20T23:02:11.118 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34757.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34757.log.gz 2026-02-20T23:02:11.119 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49010.log 2026-02-20T23:02:11.119 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55944.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55944.log.gz 2026-02-20T23:02:11.120 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32457.log 2026-02-20T23:02:11.120 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49010.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49010.log.gz 2026-02-20T23:02:11.120 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47893.log 2026-02-20T23:02:11.121 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32457.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32457.log.gz 2026-02-20T23:02:11.121 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39034.log 2026-02-20T23:02:11.122 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47893.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47893.log.gz 2026-02-20T23:02:11.122 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46174.log 2026-02-20T23:02:11.123 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39034.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39034.log.gz 2026-02-20T23:02:11.123 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55866.log 2026-02-20T23:02:11.125 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46174.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46174.log.gz 2026-02-20T23:02:11.125 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20090.log 2026-02-20T23:02:11.125 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55866.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55866.log.gz 2026-02-20T23:02:11.127 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35636.log 2026-02-20T23:02:11.127 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.20090.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.20090.log.gz 2026-02-20T23:02:11.127 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40593.log 2026-02-20T23:02:11.127 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35636.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35636.log.gz 2026-02-20T23:02:11.128 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47645.log 2026-02-20T23:02:11.128 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40593.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40593.log.gz 2026-02-20T23:02:11.128 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28617.log 2026-02-20T23:02:11.129 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47645.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47645.log.gz 2026-02-20T23:02:11.129 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49862.log 2026-02-20T23:02:11.129 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28617.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.28617.log.gz 2026-02-20T23:02:11.130 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34555.log 2026-02-20T23:02:11.130 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.49862.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49862.log.gz 2026-02-20T23:02:11.131 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52708.log 2026-02-20T23:02:11.146 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34555.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.55840.log 2026-02-20T23:02:11.147 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52708.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52708.log.gz 2026-02-20T23:02:11.154 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47500.log 2026-02-20T23:02:11.154 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.55840.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.55840.log.gz 2026-02-20T23:02:11.156 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32499.log.gz 2026-02-20T23:02:11.156 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45613.log 2026-02-20T23:02:11.157 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47500.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47500.log.gz 2026-02-20T23:02:11.157 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22360.log 2026-02-20T23:02:11.158 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45613.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.45613.log.gz 2026-02-20T23:02:11.158 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27944.log 2026-02-20T23:02:11.162 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22360.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30874.log.gz 2026-02-20T23:02:11.162 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23604.log 2026-02-20T23:02:11.164 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27944.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.22360.log.gz 2026-02-20T23:02:11.165 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33638.log 2026-02-20T23:02:11.166 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23604.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27944.log.gz 2026-02-20T23:02:11.169 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36423.log 2026-02-20T23:02:11.169 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.23604.log.gz 2026-02-20T23:02:11.169 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33638.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33638.log.gz 2026-02-20T23:02:11.169 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42012.log 2026-02-20T23:02:11.169 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36423.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26713.log 2026-02-20T23:02:11.170 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42012.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36423.log.gz 2026-02-20T23:02:11.170 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42012.log.gz 2026-02-20T23:02:11.170 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32951.log 2026-02-20T23:02:11.170 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26713.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51457.log 2026-02-20T23:02:11.170 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32951.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32951.log.gz 2026-02-20T23:02:11.173 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50872.log 2026-02-20T23:02:11.173 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51457.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51457.log.gz 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53587.log 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.26713.log.gz 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50872.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50872.log.gz 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27174.log 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53587.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53587.log.gz 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36596.log 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27174.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55023.log 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36596.log: 90.2% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36596.log.gz 2026-02-20T23:02:11.174 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.27174.log.gz 2026-02-20T23:02:11.175 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50759.log 2026-02-20T23:02:11.177 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55023.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55023.log.gz 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51412.log 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50759.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37943.log 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50759.log.gz 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51412.log: 87.0%gzip -- replaced with /var/log/ceph/cluster1-client.mirror.51412.log.gz 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42237.log 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37943.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37943.log.gz 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19690.log 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19131.log 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42237.log: /var/log/ceph/cluster1-client.mirror.19690.log: 77.6% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42237.log.gz 2026-02-20T23:02:11.178 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.19690.log.gz 2026-02-20T23:02:11.179 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29178.log 2026-02-20T23:02:11.182 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41631.log 2026-02-20T23:02:11.182 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19131.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19131.log.gz 2026-02-20T23:02:11.182 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29178.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25679.log 2026-02-20T23:02:11.182 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29178.log.gz 2026-02-20T23:02:11.182 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41631.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41631.log.gz 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51659.log 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25679.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33596.log 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr: 88.9%/var/log/ceph/cluster1-client.mirror.51659.log: -- replaced with /var/log/ceph/cluster2-client.mirror.25679.log.gz 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51659.log.gz 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44245.log 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33596.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42645.log 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33596.log.gz/var/log/ceph/cluster2-client.mirror.44245.log: 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25653.log 2026-02-20T23:02:11.183 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44245.log.gz 2026-02-20T23:02:11.184 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42645.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34555.log.gz 2026-02-20T23:02:11.185 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32615.log 2026-02-20T23:02:11.185 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32556.log 2026-02-20T23:02:11.188 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25653.log: /var/log/ceph/cluster2-client.mirror.32615.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.25653.log.gz 2026-02-20T23:02:11.188 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32615.log.gz 2026-02-20T23:02:11.188 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.22692.log.gz 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29576.log 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32556.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33210.log 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39894.log 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29576.log: -- replaced with /var/log/ceph/cluster2-client.mirror.32556.log.gz 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29576.log.gz 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33210.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46397.log 2026-02-20T23:02:11.189 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39894.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30360.log 2026-02-20T23:02:11.190 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46397.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39894.log.gz 2026-02-20T23:02:11.190 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.46397.log.gz 2026-02-20T23:02:11.195 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42645.log.gz 2026-02-20T23:02:11.196 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24323.log 2026-02-20T23:02:11.196 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.30360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53841.log 2026-02-20T23:02:11.197 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24323.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26996.log 2026-02-20T23:02:11.197 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53841.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.24323.log.gz 2026-02-20T23:02:11.197 INFO:teuthology.orchestra.run.trial107.stderr: 95.2% -- replaced with /var/log/ceph/cluster1-client.mirror.30360.log.gz 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53803.log 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26996.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28577.log 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53841.log.gz 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.21001.log.gz 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53803.log: -- replaced with /var/log/ceph/cluster1-client.mirror.26996.log.gz 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38035.log 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28577.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37782.log 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38035.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28577.log.gz 2026-02-20T23:02:11.200 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43835.log 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38035.log.gz 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53803.log.gz 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52132.log 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37782.log: /var/log/ceph/cluster1-client.mirror.43835.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32050.log 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52132.log: 87.8% 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.37782.log.gz 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.43835.log.gz 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43434.log 2026-02-20T23:02:11.201 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.52132.log.gz 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37923.log 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43434.log: gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32050.log.gz 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.37923.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30471.log 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25412.log 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2.audit.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.43434.log.gz 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30471.log: 92.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47686.log 2026-02-20T23:02:11.203 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37923.log.gz 2026-02-20T23:02:11.204 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25412.log: 92.7% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2026-02-20T23:02:11.205 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.25412.log.gz 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38817.log 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49361.log 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38817.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47686.log.gz 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52318.log 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38817.log.gz 2026-02-20T23:02:11.215 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37628.log 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49361.log: /var/log/ceph/cluster2-client.mirror.52318.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48410.log 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37628.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52318.log.gz 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49361.log.gz 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37628.log.gz 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50827.log 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48410.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49816.log 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47107.log 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50827.log.gz 2026-02-20T23:02:11.218 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49816.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42413.log 2026-02-20T23:02:11.219 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47107.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48410.log.gz 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49816.log.gz 2026-02-20T23:02:11.219 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.220 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29495.log 2026-02-20T23:02:11.221 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27089.log 2026-02-20T23:02:11.221 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42413.log: /var/log/ceph/cluster2-client.mirror.29495.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47107.log.gz 2026-02-20T23:02:11.221 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29495.log.gz 2026-02-20T23:02:11.221 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40042.log 2026-02-20T23:02:11.221 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27089.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42413.log.gz 2026-02-20T23:02:11.223 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27089.log.gz 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55632.log 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40042.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47831.log 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.40042.log.gz/var/log/ceph/cluster2-client.mirror.55632.log: 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55632.log.gz 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45493.log 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24121.log 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47831.log: /var/log/ceph/cluster2-client.mirror.45493.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48286.log 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45493.log.gz/var/log/ceph/cluster2-client.mirror.24121.log: 2026-02-20T23:02:11.226 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47831.log.gz 2026-02-20T23:02:11.227 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.24121.log.gz 2026-02-20T23:02:11.227 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54979.log 2026-02-20T23:02:11.227 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48286.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54855.log 2026-02-20T23:02:11.227 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28697.log 2026-02-20T23:02:11.229 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54979.log: /var/log/ceph/cluster2-client.mirror.54855.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54855.log.gz 2026-02-20T23:02:11.229 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54979.log.gz 2026-02-20T23:02:11.229 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48286.log.gz 2026-02-20T23:02:11.229 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24301.log 2026-02-20T23:02:11.229 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28697.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26948.log 2026-02-20T23:02:11.230 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24301.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.28697.log.gz 2026-02-20T23:02:11.230 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32092.log 2026-02-20T23:02:11.230 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34372.log 2026-02-20T23:02:11.230 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26948.log: /var/log/ceph/cluster2-client.mirror.32092.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.26948.log.gz 2026-02-20T23:02:11.230 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38640.log 2026-02-20T23:02:11.232 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34372.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34372.log.gz 2026-02-20T23:02:11.232 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20277.log 2026-02-20T23:02:11.232 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38640.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.24301.log.gz 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38640.log.gz 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33894.log 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41361.log 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20277.log: /var/log/ceph/cluster1-client.mirror.33894.log: 56.0% -- replaced with /var/log/ceph/cluster2-client.mirror.20277.log.gz 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33894.log.gz 2026-02-20T23:02:11.233 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31892.log 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29396.log 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41361.log: /var/log/ceph/cluster2-client.mirror.31892.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.41361.log.gz 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31892.log.gz 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35439.log 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29396.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39801.log 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35439.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35439.log.gz 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29396.log.gz 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.19582.log 2026-02-20T23:02:11.235 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39801.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35526.log 2026-02-20T23:02:11.236 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39801.log.gz 2026-02-20T23:02:11.237 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.3.19582.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.3.19582.log.gz 2026-02-20T23:02:11.239 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32171.log.gz 2026-02-20T23:02:11.249 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32815.log.gz 2026-02-20T23:02:11.250 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27924.log 2026-02-20T23:02:11.250 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53010.log 2026-02-20T23:02:11.251 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27924.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38889.log 2026-02-20T23:02:11.251 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35526.log.gz 2026-02-20T23:02:11.251 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53010.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18896.log 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53010.log.gz 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30690.log 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38889.log: /var/log/ceph/cluster2-client.admin.18896.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32914.log 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18896.log.gz 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30690.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.38889.log.gz 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55367.log 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30690.log.gz 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32914.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32914.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51052.log 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.253 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55367.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30611.log 2026-02-20T23:02:11.254 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55367.log.gz 2026-02-20T23:02:11.254 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52504.log 2026-02-20T23:02:11.255 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51052.log: /var/log/ceph/cluster2-client.mirror.30611.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49175.log 2026-02-20T23:02:11.255 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51052.log.gz 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27924.log.gz 2026-02-20T23:02:11.255 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52504.log: 2026-02-20T23:02:11.255 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30611.log.gz 2026-02-20T23:02:11.255 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.19582.log 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49175.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28139.log 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.19582.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49175.log.gz 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36689.log 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr: 78.7% -- replaced with /var/log/ceph/cluster2-client.mirror.19582.log.gz 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34172.log 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28139.log: /var/log/ceph/cluster2-client.mirror.36689.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31190.log 2026-02-20T23:02:11.256 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34172.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.36689.log.gz 2026-02-20T23:02:11.257 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34172.log.gz 2026-02-20T23:02:11.257 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32228.log 2026-02-20T23:02:11.257 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31190.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30078.log 2026-02-20T23:02:11.258 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32228.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32228.log.gz 2026-02-20T23:02:11.259 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28139.log.gz 2026-02-20T23:02:11.266 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32635.log 2026-02-20T23:02:11.266 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30078.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52191.log 2026-02-20T23:02:11.267 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32635.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32635.log.gz 2026-02-20T23:02:11.267 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39151.log 2026-02-20T23:02:11.268 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52191.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52191.log.gz 2026-02-20T23:02:11.268 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47169.log 2026-02-20T23:02:11.269 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39151.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39151.log.gz 2026-02-20T23:02:11.270 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45770.log 2026-02-20T23:02:11.271 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47169.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47169.log.gz 2026-02-20T23:02:11.272 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38793.log 2026-02-20T23:02:11.272 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45770.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45770.log.gz 2026-02-20T23:02:11.273 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48720.log 2026-02-20T23:02:11.273 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38793.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38793.log.gz 2026-02-20T23:02:11.274 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26670.log 2026-02-20T23:02:11.275 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48720.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48720.log.gz 2026-02-20T23:02:11.276 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31637.log 2026-02-20T23:02:11.280 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.26670.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.26670.log.gz 2026-02-20T23:02:11.280 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.22461.log 2026-02-20T23:02:11.281 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31637.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31637.log.gz 2026-02-20T23:02:11.281 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28888.log 2026-02-20T23:02:11.282 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.22461.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.22461.log.gz 2026-02-20T23:02:11.282 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47789.log 2026-02-20T23:02:11.283 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28888.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.28888.log.gz 2026-02-20T23:02:11.283 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.19532.log 2026-02-20T23:02:11.285 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47789.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47789.log.gz 2026-02-20T23:02:11.285 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31696.log 2026-02-20T23:02:11.286 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.19532.log: 76.6% -- replaced with /var/log/ceph/cluster2-client.mirror.19532.log.gz 2026-02-20T23:02:11.286 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24682.log 2026-02-20T23:02:11.298 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31696.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37371.log 2026-02-20T23:02:11.301 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.24682.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.24682.log.gz 2026-02-20T23:02:11.318 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47997.log 2026-02-20T23:02:11.318 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37371.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37371.log.gz 2026-02-20T23:02:11.329 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30226.log.gz 2026-02-20T23:02:11.329 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27330.log 2026-02-20T23:02:11.330 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47997.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47997.log.gz 2026-02-20T23:02:11.330 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30321.log 2026-02-20T23:02:11.331 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27330.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.27330.log.gz 2026-02-20T23:02:11.332 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49927.log 2026-02-20T23:02:11.332 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30321.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30321.log.gz 2026-02-20T23:02:11.332 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41289.log 2026-02-20T23:02:11.333 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.49927.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49927.log.gz 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30078.log.gz 2026-02-20T23:02:11.333 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53696.log 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41289.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49154.log 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53696.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41289.log.gz 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28073.log 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53696.log.gz 2026-02-20T23:02:11.335 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49154.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40155.log 2026-02-20T23:02:11.336 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28073.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49154.log.gz 2026-02-20T23:02:11.336 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38984.log 2026-02-20T23:02:11.337 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40155.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40155.log.gz 2026-02-20T23:02:11.337 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41402.log 2026-02-20T23:02:11.338 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38984.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38984.log.gz 2026-02-20T23:02:11.339 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.28073.log.gz 2026-02-20T23:02:11.339 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46210.log 2026-02-20T23:02:11.339 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41402.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52663.log 2026-02-20T23:02:11.341 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46210.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46210.log.gz 2026-02-20T23:02:11.341 INFO:teuthology.orchestra.run.trial107.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.41402.log.gz 2026-02-20T23:02:11.341 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38217.log 2026-02-20T23:02:11.341 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52663.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41743.log 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38217.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52663.log.gz 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33052.log.gz 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28757.log 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41743.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29027.log 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28757.log: 87.8% 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.41743.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.38217.log.gz 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28757.log.gz 2026-02-20T23:02:11.343 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35246.log 2026-02-20T23:02:11.344 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37878.log 2026-02-20T23:02:11.352 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35246.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51974.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29027.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.29894.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37878.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25751.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35246.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51974.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37878.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32835.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51974.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25751.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41651.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 88.8%/var/log/ceph/cluster2-client.mirror.32835.log: -- replaced with /var/log/ceph/cluster1-client.mirror.25751.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51704.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32835.log.gz 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45021.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51704.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54386.log 2026-02-20T23:02:11.353 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51704.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.45021.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40751.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54386.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33985.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.41651.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 89.8%/var/log/ceph/cluster2-client.mirror.40751.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45021.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54386.log.gz 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40751.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.30360.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36134.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.0.30360.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.56048.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33985.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36134.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40296.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36134.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.56048.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38131.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.56048.log.gz 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40296.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33321.log 2026-02-20T23:02:11.354 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40296.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38131.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28737.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33321.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.38131.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40639.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: 79.9%/var/log/ceph/cluster2-client.mirror.28737.log: -- replaced with /var/log/ceph/cluster2-client.mirror.33321.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.28737.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51074.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40639.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49320.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55682.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51074.log.gz/var/log/ceph/cluster2-client.mirror.49320.log: 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40639.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49320.log.gzgzip 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24420.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55682.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26972.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55682.log.gz 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24420.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23955.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26972.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36444.log 2026-02-20T23:02:11.355 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23955.log: 91.7% 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.26972.log.gz 2026-02-20T23:02:11.356 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.24420.log.gz 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38616.log 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36444.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36444.log.gz 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29007.log 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38616.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38616.log.gz 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.44541.log 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29007.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29007.log.gz 2026-02-20T23:02:11.357 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39251.log 2026-02-20T23:02:11.358 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.2.44541.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32013.log.gz 2026-02-20T23:02:11.358 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35484.log 2026-02-20T23:02:11.358 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39251.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30512.log 2026-02-20T23:02:11.358 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39251.log.gz 2026-02-20T23:02:11.359 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35484.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18976.log 2026-02-20T23:02:11.360 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30512.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30512.log.gz 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35484.log.gz 2026-02-20T23:02:11.360 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.360 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53331.log 2026-02-20T23:02:11.360 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44999.log 2026-02-20T23:02:11.360 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18976.log: /var/log/ceph/cluster1-client.mirror.53331.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18976.log.gz 2026-02-20T23:02:11.361 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53331.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42511.log 2026-02-20T23:02:11.361 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.361 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44999.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.52526.log 2026-02-20T23:02:11.362 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42511.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.42511.log.gz 2026-02-20T23:02:11.362 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44999.log.gz 2026-02-20T23:02:11.362 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50602.log 2026-02-20T23:02:11.363 INFO:teuthology.orchestra.run.trial107.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.44541.log.gz 2026-02-20T23:02:11.363 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40090.log 2026-02-20T23:02:11.363 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.1.52526.log: /var/log/ceph/cluster1-client.mirror.50602.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19304.log 2026-02-20T23:02:11.363 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50602.log.gz 2026-02-20T23:02:11.364 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.30360.log.gz 2026-02-20T23:02:11.364 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40090.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40090.log.gz 2026-02-20T23:02:11.367 INFO:teuthology.orchestra.run.trial107.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.1.52526.log.gz 2026-02-20T23:02:11.367 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34059.log 2026-02-20T23:02:11.368 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40018.log 2026-02-20T23:02:11.368 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19304.log: /var/log/ceph/cluster1-client.mirror.34059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29872.log 2026-02-20T23:02:11.368 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19304.log.gz 2026-02-20T23:02:11.368 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.40018.log: -- replaced with /var/log/ceph/cluster1-client.mirror.34059.log.gz 2026-02-20T23:02:11.368 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44910.log 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.40018.log.gz 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29872.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31753.log 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44910.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28477.log 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29872.log.gz 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44910.log.gz/var/log/ceph/cluster2-client.mirror.31753.log: 2026-02-20T23:02:11.369 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55302.log 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.31753.log.gzgzip 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32993.log 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28477.log: /var/log/ceph/cluster1-client.mirror.55302.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31269.log 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55302.log.gz 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32993.log: 88.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20209.log 2026-02-20T23:02:11.370 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.28477.log.gz 2026-02-20T23:02:11.371 INFO:teuthology.orchestra.run.trial107.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32993.log.gz 2026-02-20T23:02:11.371 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31269.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44406.log 2026-02-20T23:02:11.371 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20209.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48162.log 2026-02-20T23:02:11.371 INFO:teuthology.orchestra.run.trial107.stderr: 66.1% -- replaced with /var/log/ceph/cluster2-client.mirror.20209.log.gz 2026-02-20T23:02:11.371 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44406.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46726.log 2026-02-20T23:02:11.372 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48162.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44406.log.gz 2026-02-20T23:02:11.373 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51164.log 2026-02-20T23:02:11.373 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46726.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48162.log.gz 2026-02-20T23:02:11.373 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.46726.log.gz 2026-02-20T23:02:11.374 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.12203.log 2026-02-20T23:02:11.374 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27375.log 2026-02-20T23:02:11.376 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51164.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35287.log 2026-02-20T23:02:11.376 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/ceph.tmp-client.admin.12203.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51164.log.gz 2026-02-20T23:02:11.376 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.12203.log.gz 2026-02-20T23:02:11.376 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23803.log 2026-02-20T23:02:11.376 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37693.log 2026-02-20T23:02:11.377 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35287.log: /var/log/ceph/cluster1-client.mirror.23803.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.23803.log.gz 2026-02-20T23:02:11.377 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35287.log.gz 2026-02-20T23:02:11.377 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.24258.log 2026-02-20T23:02:11.377 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51097.log 2026-02-20T23:02:11.378 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37693.log: /var/log/ceph/cluster1-client.mirror.24258.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.27375.log.gz 2026-02-20T23:02:11.379 INFO:teuthology.orchestra.run.trial107.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37693.log.gz 2026-02-20T23:02:11.379 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42487.log 2026-02-20T23:02:11.379 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51097.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43978.log 2026-02-20T23:02:11.379 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51097.log.gz 2026-02-20T23:02:11.380 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42487.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45280.log 2026-02-20T23:02:11.380 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43978.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42487.log.gz 2026-02-20T23:02:11.380 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55323.log 2026-02-20T23:02:11.380 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43978.log.gz 2026-02-20T23:02:11.381 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45280.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.24258.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19054.log 2026-02-20T23:02:11.381 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.381 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55323.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45280.log.gz 2026-02-20T23:02:11.382 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54872.log 2026-02-20T23:02:11.382 INFO:teuthology.orchestra.run.trial107.stderr: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55323.log.gz 2026-02-20T23:02:11.382 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52792.log 2026-02-20T23:02:11.382 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19054.log: /var/log/ceph/cluster2-client.mirror.54872.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51502.log 2026-02-20T23:02:11.382 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.19054.log.gz 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52792.log: 79.1% -- replaced with /var/log/ceph/cluster1-client.mirror.52792.log.gz 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr: 89.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36793.log 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.54872.log.gz 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51502.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37652.log 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36793.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51502.log.gz 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32477.log 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.37652.log: -- replaced with /var/log/ceph/cluster2-client.mirror.36793.log.gz 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43032.log 2026-02-20T23:02:11.384 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37652.log.gz 2026-02-20T23:02:11.385 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39434.log 2026-02-20T23:02:11.385 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32477.log: /var/log/ceph/cluster2-client.mirror.43032.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32477.log.gz 2026-02-20T23:02:11.385 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43032.log.gz 2026-02-20T23:02:11.385 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49609.log 2026-02-20T23:02:11.385 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49671.log 2026-02-20T23:02:11.386 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39434.log: /var/log/ceph/cluster2-client.mirror.49609.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.19581.log 2026-02-20T23:02:11.386 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49671.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49609.log.gz 2026-02-20T23:02:11.386 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39434.log.gz 2026-02-20T23:02:11.386 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.49671.log.gz 2026-02-20T23:02:11.387 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52729.log 2026-02-20T23:02:11.387 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47665.log 2026-02-20T23:02:11.387 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.19581.log: /var/log/ceph/cluster2-client.mirror.52729.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.52729.log.gz 2026-02-20T23:02:11.388 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33072.log 2026-02-20T23:02:11.389 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47665.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47665.log.gz 2026-02-20T23:02:11.389 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36976.log 2026-02-20T23:02:11.390 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33072.log: 80.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38107.log 2026-02-20T23:02:11.390 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33072.log.gz 2026-02-20T23:02:11.391 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36976.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54430.log 2026-02-20T23:02:11.391 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.19581.log.gz 2026-02-20T23:02:11.391 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38107.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36976.log.gz 2026-02-20T23:02:11.391 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37902.log 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54430.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38107.log.gz 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37436.log 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.54430.log.gz 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37902.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24964.log 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.37902.log.gz 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36182.log 2026-02-20T23:02:11.392 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37436.log: /var/log/ceph/cluster2-client.mirror.24964.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53675.log 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36182.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.36182.log.gz 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.24964.log.gz 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28030.log 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37436.log.gz/var/log/ceph/cluster1-client.mirror.53675.log: 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.20107.log 2026-02-20T23:02:11.393 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53675.log.gz 2026-02-20T23:02:11.394 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28030.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44145.log 2026-02-20T23:02:11.394 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55000.log 2026-02-20T23:02:11.394 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.20107.log: /var/log/ceph/cluster2-client.mirror.44145.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.20107.log.gz 2026-02-20T23:02:11.395 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23582.log 2026-02-20T23:02:11.395 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.55000.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55000.log.gz 2026-02-20T23:02:11.395 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44145.log.gz 2026-02-20T23:02:11.396 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43701.log 2026-02-20T23:02:11.396 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54365.log 2026-02-20T23:02:11.397 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23582.log: /var/log/ceph/cluster2-client.mirror.43701.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43701.log.gz 2026-02-20T23:02:11.397 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.28030.log.gz 2026-02-20T23:02:11.397 INFO:teuthology.orchestra.run.trial107.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.23582.log.gz 2026-02-20T23:02:11.397 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26625.log 2026-02-20T23:02:11.398 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54365.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37323.log 2026-02-20T23:02:11.398 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26625.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30449.log 2026-02-20T23:02:11.398 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54365.log.gz 2026-02-20T23:02:11.398 INFO:teuthology.orchestra.run.trial107.stderr: 89.5% -- replaced with /var/log/ceph/cluster1-client.mirror.26625.log.gz 2026-02-20T23:02:11.399 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37323.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33437.log 2026-02-20T23:02:11.399 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30449.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37323.log.gz 2026-02-20T23:02:11.399 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52156.log 2026-02-20T23:02:11.399 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30449.log.gz 2026-02-20T23:02:11.399 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32872.log 2026-02-20T23:02:11.400 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33437.log: /var/log/ceph/cluster1-client.mirror.52156.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36644.log 2026-02-20T23:02:11.400 INFO:teuthology.orchestra.run.trial107.stderr: 41.6% -- replaced with /var/log/ceph/cluster1-client.mirror.52156.log.gz 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32872.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33437.log.gz 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.26832.log 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32872.log.gz 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36644.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39968.log 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36644.log.gz 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31439.log 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.26832.log: /var/log/ceph/cluster2-client.mirror.39968.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29593.log 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr: 88.3%/var/log/ceph/cluster2-client.mirror.31439.log: -- replaced with /var/log/ceph/cluster1-client.admin.26832.log.gz 2026-02-20T23:02:11.401 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39968.log.gz 2026-02-20T23:02:11.414 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43393.log 2026-02-20T23:02:11.414 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23283.log 2026-02-20T23:02:11.414 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43393.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.29593.log.gz 2026-02-20T23:02:11.415 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.43393.log.gz 2026-02-20T23:02:11.415 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25477.log 2026-02-20T23:02:11.415 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23283.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27068.log 2026-02-20T23:02:11.415 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.23283.log.gz 2026-02-20T23:02:11.416 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.25477.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53138.log 2026-02-20T23:02:11.416 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.25477.log.gz 2026-02-20T23:02:11.417 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27068.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41382.log 2026-02-20T23:02:11.417 INFO:teuthology.orchestra.run.trial107.stderr: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27068.log.gz 2026-02-20T23:02:11.417 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53138.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33655.log 2026-02-20T23:02:11.418 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53138.log.gz 2026-02-20T23:02:11.418 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41382.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34479.log 2026-02-20T23:02:11.418 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33655.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41382.log.gz 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33655.log.gz 2026-02-20T23:02:11.418 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.418 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23161.log 2026-02-20T23:02:11.419 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34479.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33109.log 2026-02-20T23:02:11.419 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23161.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34479.log.gz 2026-02-20T23:02:11.420 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48762.log 2026-02-20T23:02:11.420 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33109.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.23161.log.gz 2026-02-20T23:02:11.420 INFO:teuthology.orchestra.run.trial107.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31775.log 2026-02-20T23:02:11.420 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.33109.log.gz 2026-02-20T23:02:11.420 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48762.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.25775.log 2026-02-20T23:02:11.421 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31775.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48762.log.gz 2026-02-20T23:02:11.421 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46308.log 2026-02-20T23:02:11.422 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.25775.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.25775.log.gz 2026-02-20T23:02:11.422 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51142.log 2026-02-20T23:02:11.423 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46308.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46308.log.gz 2026-02-20T23:02:11.423 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42463.log 2026-02-20T23:02:11.424 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51142.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51142.log.gz 2026-02-20T23:02:11.424 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34152.log 2026-02-20T23:02:11.425 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42463.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42463.log.gz 2026-02-20T23:02:11.425 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35504.log 2026-02-20T23:02:11.427 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34152.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34152.log.gz 2026-02-20T23:02:11.427 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35367.log 2026-02-20T23:02:11.429 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35504.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35504.log.gz 2026-02-20T23:02:11.429 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26224.log 2026-02-20T23:02:11.430 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35367.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35367.log.gz 2026-02-20T23:02:11.430 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31934.log 2026-02-20T23:02:11.431 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26224.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.26224.log.gz 2026-02-20T23:02:11.431 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.15974.log 2026-02-20T23:02:11.438 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31934.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54616.log 2026-02-20T23:02:11.442 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.15974.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.15974.log.gz 2026-02-20T23:02:11.454 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46477.log 2026-02-20T23:02:11.455 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32894.log.gz 2026-02-20T23:02:11.455 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54616.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54616.log.gz 2026-02-20T23:02:11.455 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44518.log 2026-02-20T23:02:11.455 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46477.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46477.log.gz 2026-02-20T23:02:11.456 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22511.log 2026-02-20T23:02:11.456 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44518.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44518.log.gz 2026-02-20T23:02:11.456 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50782.log 2026-02-20T23:02:11.457 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.22511.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.22511.log.gz 2026-02-20T23:02:11.457 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33804.log 2026-02-20T23:02:11.458 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50782.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50782.log.gz 2026-02-20T23:02:11.458 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52297.log 2026-02-20T23:02:11.460 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.33804.log: 95.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33804.log.gz 2026-02-20T23:02:11.460 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34505.log 2026-02-20T23:02:11.461 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52297.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52297.log.gz 2026-02-20T23:02:11.461 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48431.log 2026-02-20T23:02:11.462 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34505.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34505.log.gz 2026-02-20T23:02:11.462 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20336.log 2026-02-20T23:02:11.463 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48431.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.48431.log.gz 2026-02-20T23:02:11.463 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52619.log 2026-02-20T23:02:11.464 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20336.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.20336.log.gz 2026-02-20T23:02:11.464 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44541.log 2026-02-20T23:02:11.465 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52619.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52619.log.gz 2026-02-20T23:02:11.465 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33576.log 2026-02-20T23:02:11.466 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44541.log: 94.6% -- replaced with /var/log/ceph/cluster1-client.mirror.44541.log.gz 2026-02-20T23:02:11.466 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49464.log 2026-02-20T23:02:11.467 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33576.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33576.log.gz 2026-02-20T23:02:11.467 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27234.log 2026-02-20T23:02:11.469 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49464.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49464.log.gz 2026-02-20T23:02:11.469 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52362.log 2026-02-20T23:02:11.470 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27234.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.27234.log.gz 2026-02-20T23:02:11.470 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54637.log 2026-02-20T23:02:11.471 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.52362.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52362.log.gz 2026-02-20T23:02:11.471 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.23779.log 2026-02-20T23:02:11.472 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54637.log: 88.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54637.log.gz 2026-02-20T23:02:11.472 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37556.log 2026-02-20T23:02:11.473 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.23779.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.23779.log.gz 2026-02-20T23:02:11.473 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.27901.log 2026-02-20T23:02:11.474 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37556.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37556.log.gz 2026-02-20T23:02:11.474 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20141.log 2026-02-20T23:02:11.477 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.27901.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31111.log.gz 2026-02-20T23:02:11.477 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38590.log 2026-02-20T23:02:11.478 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20141.log: 63.5% -- replaced with /var/log/ceph/cluster2-client.mirror.20141.log.gz 2026-02-20T23:02:11.478 INFO:teuthology.orchestra.run.trial107.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.27901.log.gz 2026-02-20T23:02:11.478 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46573.log 2026-02-20T23:02:11.479 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37299.log 2026-02-20T23:02:11.479 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38590.log: /var/log/ceph/cluster1-client.mirror.46573.log: 87.8% 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46573.log.gz 2026-02-20T23:02:11.479 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.38590.log.gz 2026-02-20T23:02:11.479 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28947.log 2026-02-20T23:02:11.480 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48534.log 2026-02-20T23:02:11.480 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37299.log.gz 2026-02-20T23:02:11.480 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28947.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46638.log 2026-02-20T23:02:11.482 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48534.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48534.log.gz 2026-02-20T23:02:11.482 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54914.log 2026-02-20T23:02:11.483 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.46638.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46638.log.gz 2026-02-20T23:02:11.484 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46132.log 2026-02-20T23:02:11.494 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.54914.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44934.log 2026-02-20T23:02:11.495 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.46132.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46132.log.gz 2026-02-20T23:02:11.497 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32092.log.gz 2026-02-20T23:02:11.497 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40320.log 2026-02-20T23:02:11.498 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44934.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44934.log.gz 2026-02-20T23:02:11.498 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27838.log 2026-02-20T23:02:11.499 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40320.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.40320.log.gz 2026-02-20T23:02:11.500 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33338.log 2026-02-20T23:02:11.503 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27838.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27838.log.gz 2026-02-20T23:02:11.503 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30011.log 2026-02-20T23:02:11.504 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33338.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33338.log.gz 2026-02-20T23:02:11.506 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.19629.log 2026-02-20T23:02:11.506 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30011.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.30011.log.gz 2026-02-20T23:02:11.506 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.19279.log 2026-02-20T23:02:11.514 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.0.19629.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.33232.log 2026-02-20T23:02:11.514 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.19279.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.19279.log.gz 2026-02-20T23:02:11.514 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.19629.log.gz 2026-02-20T23:02:11.514 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30712.log 2026-02-20T23:02:11.522 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.0.33232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18897.log 2026-02-20T23:02:11.538 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30712.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48658.log 2026-02-20T23:02:11.539 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.18897.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18897.log.gz 2026-02-20T23:02:11.539 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55199.log 2026-02-20T23:02:11.540 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48658.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48658.log.gz 2026-02-20T23:02:11.545 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26376.log.gz 2026-02-20T23:02:11.546 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30911.log 2026-02-20T23:02:11.547 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55199.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55199.log.gz 2026-02-20T23:02:11.549 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32420.log.gz 2026-02-20T23:02:11.555 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.33232.log.gz 2026-02-20T23:02:11.562 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29237.log.gz 2026-02-20T23:02:11.562 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32677.log 2026-02-20T23:02:11.565 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30911.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.19429.log 2026-02-20T23:02:11.565 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32677.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32677.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.27967.log 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30911.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.19429.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.19429.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53464.log 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44004.log 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.27967.log: /var/log/ceph/cluster1-client.mirror.53464.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50692.log 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53464.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44004.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44004.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.54320.log.gz 2026-02-20T23:02:11.566 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32398.log 2026-02-20T23:02:11.567 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50692.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30852.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.32398.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16052.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50692.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32398.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30852.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34921.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 87.7%/var/log/ceph/cluster2-client.admin.16052.log: -- replaced with /var/log/ceph/cluster2-client.mirror.30852.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50849.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16052.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34921.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.34921.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.27967.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31733.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50849.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41424.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50849.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31733.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41904.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49299.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31733.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41904.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25504.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.16177.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49299.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41424.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.25504.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.49299.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34668.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.admin.16177.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.25504.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.41904.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.16177.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53031.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42535.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34668.log: /var/log/ceph/cluster1-client.mirror.53031.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25881.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53031.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42535.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34668.log.gz 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37830.log 2026-02-20T23:02:11.575 INFO:teuthology.orchestra.run.trial107.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.42535.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42879.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37830.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55657.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.42879.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37830.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31875.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.25881.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.42879.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.55657.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.55657.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39873.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31875.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49134.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.31875.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39873.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43590.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49134.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47562.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.39873.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49134.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43590.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33740.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51524.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.33740.log: -- replaced with /var/log/ceph/cluster1-client.mirror.43590.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20175.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.33740.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.47562.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47066.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51524.log: 86.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28677.log 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51524.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20175.log: 5.8% -- replaced with /var/log/ceph/cluster2-client.mirror.20175.log.gz 2026-02-20T23:02:11.576 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47066.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30953.log 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47066.log.gz 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51344.log 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28677.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47479.log 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30953.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.28677.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40725.log 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51344.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51344.log.gz 2026-02-20T23:02:11.577 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47479.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47479.log.gz 2026-02-20T23:02:11.578 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50062.log 2026-02-20T23:02:11.578 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40725.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35747.log 2026-02-20T23:02:11.578 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50062.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28311.log 2026-02-20T23:02:11.579 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.40725.log.gz 2026-02-20T23:02:11.579 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.50062.log.gz 2026-02-20T23:02:11.579 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35747.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47872.log 2026-02-20T23:02:11.579 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28311.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43215.log 2026-02-20T23:02:11.579 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35747.log.gz 2026-02-20T23:02:11.580 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47872.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47872.log.gz 2026-02-20T23:02:11.582 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28311.log.gz 2026-02-20T23:02:11.583 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47438.log 2026-02-20T23:02:11.583 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43215.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38480.log 2026-02-20T23:02:11.584 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47438.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50579.log 2026-02-20T23:02:11.584 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43215.log.gz 2026-02-20T23:02:11.584 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.47438.log.gz 2026-02-20T23:02:11.585 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38480.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38480.log.gz 2026-02-20T23:02:11.594 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32272.log 2026-02-20T23:02:11.594 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48741.log 2026-02-20T23:02:11.594 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50579.log.gz/var/log/ceph/cluster1-client.mirror.32272.log: 2026-02-20T23:02:11.594 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37106.log 2026-02-20T23:02:11.595 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35562.log 2026-02-20T23:02:11.595 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48741.log: /var/log/ceph/cluster1-client.mirror.37106.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.37106.log.gz 2026-02-20T23:02:11.595 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% 95.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48741.log.gz 2026-02-20T23:02:11.595 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.32272.log.gz 2026-02-20T23:02:11.596 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47252.log 2026-02-20T23:02:11.596 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35562.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.28840.log 2026-02-20T23:02:11.596 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47252.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37082.log 2026-02-20T23:02:11.596 INFO:teuthology.orchestra.run.trial107.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35562.log.gz 2026-02-20T23:02:11.596 INFO:teuthology.orchestra.run.trial107.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.47252.log.gz 2026-02-20T23:02:11.597 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.28840.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28925.log 2026-02-20T23:02:11.597 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.28840.log.gz/var/log/ceph/cluster1-client.mirror.37082.log: 2026-02-20T23:02:11.597 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39299.log 2026-02-20T23:02:11.597 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.37082.log.gz 2026-02-20T23:02:11.598 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28925.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30167.log 2026-02-20T23:02:11.598 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.28925.log.gz 2026-02-20T23:02:11.598 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.39299.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26354.log 2026-02-20T23:02:11.598 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.39299.log.gz 2026-02-20T23:02:11.599 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40409.log 2026-02-20T23:02:11.599 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30167.log: /var/log/ceph/cluster2-client.mirror.26354.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42147.log 2026-02-20T23:02:11.599 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40409.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.30167.log.gz 2026-02-20T23:02:11.599 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.26354.log.gz 2026-02-20T23:02:11.599 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49994.log 2026-02-20T23:02:11.600 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43080.log 2026-02-20T23:02:11.600 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42147.log: /var/log/ceph/cluster1-client.mirror.49994.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.40409.log.gz 2026-02-20T23:02:11.600 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49994.log.gz 2026-02-20T23:02:11.600 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42147.log.gz 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41717.log 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43080.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.25456.log 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.43080.log.gz 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.41717.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48803.log 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50984.log 2026-02-20T23:02:11.601 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29146.log.gz 2026-02-20T23:02:11.602 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.25456.log: /var/log/ceph/cluster2-client.mirror.48803.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41717.log.gz 2026-02-20T23:02:11.602 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.29793.log 2026-02-20T23:02:11.602 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48803.log.gz 2026-02-20T23:02:11.602 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50984.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50219.log 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50984.log.gz 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.29793.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53116.log 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50219.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.29793.log.gz 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24903.log 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50219.log.gz 2026-02-20T23:02:11.603 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53116.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50647.log 2026-02-20T23:02:11.604 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24903.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36668.log 2026-02-20T23:02:11.604 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50647.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.24903.log.gz 2026-02-20T23:02:11.604 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50647.log.gz 2026-02-20T23:02:11.605 INFO:teuthology.orchestra.run.trial107.stderr: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53116.log.gz 2026-02-20T23:02:11.605 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26649.log 2026-02-20T23:02:11.606 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43492.log 2026-02-20T23:02:11.606 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26649.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36668.log.gz 2026-02-20T23:02:11.606 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44862.log 2026-02-20T23:02:11.606 INFO:teuthology.orchestra.run.trial107.stderr: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.26649.log.gz 2026-02-20T23:02:11.606 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43492.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38841.log 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.44862.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49650.log 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43492.log.gz 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38841.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44862.log.gz 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.38841.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34324.log 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.607 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49650.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35588.log 2026-02-20T23:02:11.608 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49030.log 2026-02-20T23:02:11.608 INFO:teuthology.orchestra.run.trial107.stderr: 87.9%/var/log/ceph/cluster2-client.mirror.35588.log: -- replaced with /var/log/ceph/cluster1-client.mirror.34324.log.gz 2026-02-20T23:02:11.608 INFO:teuthology.orchestra.run.trial107.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35588.log.gz 2026-02-20T23:02:11.608 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47913.log 2026-02-20T23:02:11.609 INFO:teuthology.orchestra.run.trial107.stderr: 90.5%/var/log/ceph/cluster2-client.mirror.49030.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49650.log.gz 2026-02-20T23:02:11.609 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26553.log 2026-02-20T23:02:11.609 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47913.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43659.log 2026-02-20T23:02:11.609 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26553.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31190.log.gz 2026-02-20T23:02:11.610 INFO:teuthology.orchestra.run.trial107.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.26553.log.gz 2026-02-20T23:02:11.610 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.49030.log.gz 2026-02-20T23:02:11.610 INFO:teuthology.orchestra.run.trial107.stderr: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47913.log.gz 2026-02-20T23:02:11.610 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30591.log 2026-02-20T23:02:11.611 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43659.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34435.log 2026-02-20T23:02:11.611 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30591.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.26922.log 2026-02-20T23:02:11.611 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34435.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39225.log 2026-02-20T23:02:11.612 INFO:teuthology.orchestra.run.trial107.stderr: 89.7%/var/log/ceph/cluster2-client.mirror.26922.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43659.log.gz 2026-02-20T23:02:11.612 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.30591.log.gz 2026-02-20T23:02:11.612 INFO:teuthology.orchestra.run.trial107.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34435.log.gz 2026-02-20T23:02:11.612 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47975.log 2026-02-20T23:02:11.612 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.39225.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.26922.log.gz 2026-02-20T23:02:11.613 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38930.log 2026-02-20T23:02:11.613 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.47975.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26733.log 2026-02-20T23:02:11.613 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.38930.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39225.log.gz 2026-02-20T23:02:11.613 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34226.log 2026-02-20T23:02:11.614 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26733.log: 90.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47975.log.gz 2026-02-20T23:02:11.614 INFO:teuthology.orchestra.run.trial107.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38930.log.gz 2026-02-20T23:02:11.614 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52462.log 2026-02-20T23:02:11.615 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34226.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38343.log 2026-02-20T23:02:11.615 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52462.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26152.log 2026-02-20T23:02:11.615 INFO:teuthology.orchestra.run.trial107.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.34226.log.gz 2026-02-20T23:02:11.615 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38343.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38343.log.gz 2026-02-20T23:02:11.615 INFO:teuthology.orchestra.run.trial107.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52462.log.gz 2026-02-20T23:02:11.616 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20243.log 2026-02-20T23:02:11.616 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54171.log 2026-02-20T23:02:11.616 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20243.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.26152.log.gz 2026-02-20T23:02:11.617 INFO:teuthology.orchestra.run.trial107.stderr: 93.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51839.log 2026-02-20T23:02:11.617 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.26733.log.gz 65.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20243.log.gz 2026-02-20T23:02:11.617 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.617 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54171.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51794.log 2026-02-20T23:02:11.617 INFO:teuthology.orchestra.run.trial107.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54171.log.gz 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51839.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.23371.log 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.51839.log.gz 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51794.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48452.log 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51794.log.gz 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.23371.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.19656.log 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34849.log 2026-02-20T23:02:11.618 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.25903.log.gz 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48452.log.gz/var/log/ceph/cluster1-client.mirror.19656.log: 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48968.log 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr: 78.2% -- replaced with /var/log/ceph/cluster1-client.mirror.19656.log.gz 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.34849.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54958.log 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48968.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34849.log.gz 2026-02-20T23:02:11.619 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35309.log 2026-02-20T23:02:11.620 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.54958.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32657.log 2026-02-20T23:02:11.620 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35309.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54958.log.gz 2026-02-20T23:02:11.620 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35309.log.gz 2026-02-20T23:02:11.620 INFO:teuthology.orchestra.run.trial107.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.48968.log.gz 2026-02-20T23:02:11.620 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28597.log 2026-02-20T23:02:11.621 INFO:teuthology.orchestra.run.trial107.stderr: 93.6%/var/log/ceph/cluster2-client.mirror.32657.log: -- replaced with /var/log/ceph/cluster2-client.mirror.23371.log.gz 2026-02-20T23:02:11.621 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.26200.log 2026-02-20T23:02:11.621 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31716.log 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.26200.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.28597.log.gzgzip 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41044.log 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.26200.log.gz 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.31716.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51277.log 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31716.log.gz 2026-02-20T23:02:11.622 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.41044.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53096.log 2026-02-20T23:02:11.623 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51277.log: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22607.log 2026-02-20T23:02:11.623 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.41044.log.gz 2026-02-20T23:02:11.623 INFO:teuthology.orchestra.run.trial107.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51277.log.gz 2026-02-20T23:02:11.623 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.53096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34194.log 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr: 88.3%/var/log/ceph/cluster1-client.mirror.22607.log: -- replaced with /var/log/ceph/cluster2-client.mirror.53096.log.gz 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36548.log 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr: 87.8%/var/log/ceph/cluster2-client.mirror.34194.log: -- replaced with /var/log/ceph/cluster1-client.mirror.22607.log.gz 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49113.log 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36548.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.24367.log 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34194.log.gz 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49113.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36548.log.gz 2026-02-20T23:02:11.624 INFO:teuthology.orchestra.run.trial107.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49113.log.gz 2026-02-20T23:02:11.625 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48617.log 2026-02-20T23:02:11.625 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.24367.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50264.log 2026-02-20T23:02:11.625 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48617.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.24367.log.gz 2026-02-20T23:02:11.625 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28437.log 2026-02-20T23:02:11.626 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.50264.log: 88.3%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49526.log 2026-02-20T23:02:11.626 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.48617.log.gz 2026-02-20T23:02:11.626 INFO:teuthology.orchestra.run.trial107.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50264.log.gz 2026-02-20T23:02:11.626 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.22973.log 2026-02-20T23:02:11.626 INFO:teuthology.orchestra.run.trial107.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.28437.log.gz 2026-02-20T23:02:11.627 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.49526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.20226.log 2026-02-20T23:02:11.627 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.22973.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51727.log 2026-02-20T23:02:11.627 INFO:teuthology.orchestra.run.trial107.stderr: 78.2% -- replaced with /var/log/ceph/cluster1-client.mirror.22973.log.gz 2026-02-20T23:02:11.627 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.20226.log: 65.3% -- replaced with /var/log/ceph/cluster2-client.mirror.20226.log.gz 2026-02-20T23:02:11.627 INFO:teuthology.orchestra.run.trial107.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49526.log.gz 2026-02-20T23:02:11.629 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33459.log.gz 2026-02-20T23:02:11.630 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.19581.log 2026-02-20T23:02:11.630 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51727.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.28096.log 2026-02-20T23:02:11.630 INFO:teuthology.orchestra.run.trial107.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51727.log.gz 2026-02-20T23:02:11.630 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.2.19581.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42373.log 2026-02-20T23:02:11.630 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.28096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36062.log 2026-02-20T23:02:11.631 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.42373.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38910.log 2026-02-20T23:02:11.631 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.36062.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42373.log.gz 2026-02-20T23:02:11.631 INFO:teuthology.orchestra.run.trial107.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36062.log.gz 2026-02-20T23:02:11.632 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35705.log 2026-02-20T23:02:11.632 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.38910.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.27705.log 2026-02-20T23:02:11.633 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.35705.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38910.log.gz 2026-02-20T23:02:11.633 INFO:teuthology.orchestra.run.trial107.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35705.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37058.log 2026-02-20T23:02:11.633 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.633 INFO:teuthology.orchestra.run.trial107.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.28096.log.gz 2026-02-20T23:02:11.634 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.1.27705.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43006.log 2026-02-20T23:02:11.634 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.37058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.40066.log 2026-02-20T23:02:11.634 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.43006.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52941.log 2026-02-20T23:02:11.634 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.37058.log.gz 2026-02-20T23:02:11.635 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.40066.log: 87.9% 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.43006.log.gz 2026-02-20T23:02:11.635 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33380.log 2026-02-20T23:02:11.635 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.40066.log.gz 2026-02-20T23:02:11.635 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.52941.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40934.log 2026-02-20T23:02:11.635 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.33380.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30832.log 2026-02-20T23:02:11.636 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.40934.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.40934.log.gz 2026-02-20T23:02:11.637 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.22310.log 2026-02-20T23:02:11.637 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.30832.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.30832.log.gz 2026-02-20T23:02:11.638 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35933.log 2026-02-20T23:02:11.638 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.admin.22310.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.22310.log.gz 2026-02-20T23:02:11.639 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45654.log 2026-02-20T23:02:11.639 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.35933.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35933.log.gz 2026-02-20T23:02:11.640 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31696.log.gz 2026-02-20T23:02:11.640 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36889.log 2026-02-20T23:02:11.640 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.45654.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34276.log 2026-02-20T23:02:11.641 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.36889.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.36889.log.gz 2026-02-20T23:02:11.641 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44979.log 2026-02-20T23:02:11.641 INFO:teuthology.orchestra.run.trial107.stderr: 92.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45654.log.gz 2026-02-20T23:02:11.641 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.34276.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.18968.log 2026-02-20T23:02:11.642 INFO:teuthology.orchestra.run.trial107.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34276.log.gz 2026-02-20T23:02:11.642 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.44979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48948.log 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr: 92.8%/var/log/ceph/cluster2-client.admin.18968.log: -- replaced with /var/log/ceph/cluster1-client.mirror.2.19581.log.gz 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.18968.log.gz 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.24452.log 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.48948.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44979.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43614.log 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:02:11.643 INFO:teuthology.orchestra.run.trial107.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.48948.log.gz 2026-02-20T23:02:11.644 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster2-client.mirror.24452.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.39624.log 2026-02-20T23:02:11.644 INFO:teuthology.orchestra.run.trial107.stderr: 90.1%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51592.log 2026-02-20T23:02:11.644 INFO:teuthology.orchestra.run.trial107.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.24452.log.gz 2026-02-20T23:02:11.644 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.43614.log: /var/log/ceph/cluster1-client.mirror.39624.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.43614.log.gz 2026-02-20T23:02:11.645 INFO:teuthology.orchestra.run.trial107.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.39624.log.gz 2026-02-20T23:02:11.645 INFO:teuthology.orchestra.run.trial107.stderr:/var/log/ceph/cluster1-client.mirror.51592.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.51592.log.gz 2026-02-20T23:02:11.701 INFO:teuthology.orchestra.run.trial107.stderr: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.1.27705.log.gz 2026-02-20T23:02:11.771 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33210.log.gz 2026-02-20T23:02:11.787 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.29736.log.gz 2026-02-20T23:02:11.817 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31934.log.gz 2026-02-20T23:02:11.858 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31439.log.gz 2026-02-20T23:02:11.887 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30471.log.gz 2026-02-20T23:02:11.888 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52504.log.gz 2026-02-20T23:02:11.892 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30953.log.gz 2026-02-20T23:02:11.942 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31269.log.gz 2026-02-20T23:02:11.989 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.25456.log.gz 2026-02-20T23:02:12.001 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30712.log.gz 2026-02-20T23:02:12.013 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33380.log.gz 2026-02-20T23:02:12.022 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.54914.log.gz 2026-02-20T23:02:12.031 INFO:teuthology.orchestra.run.trial107.stderr: 94.2% -- replaced with /var/log/ceph/cluster2-client.mirror.23955.log.gz 2026-02-20T23:02:12.074 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.28947.log.gz 2026-02-20T23:02:12.131 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.31775.log.gz 2026-02-20T23:02:12.171 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.52941.log.gz 2026-02-20T23:02:12.222 INFO:teuthology.orchestra.run.trial107.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32657.log.gz 2026-02-20T23:02:14.766 INFO:teuthology.orchestra.run.trial032.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2026-02-20T23:02:14.849 INFO:teuthology.orchestra.run.trial032.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2026-02-20T23:02:15.159 INFO:teuthology.orchestra.run.trial032.stderr: 94.5% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2026-02-20T23:02:15.160 INFO:teuthology.orchestra.run.trial032.stderr: 2026-02-20T23:02:15.160 INFO:teuthology.orchestra.run.trial032.stderr:real 0m5.242s 2026-02-20T23:02:15.160 INFO:teuthology.orchestra.run.trial032.stderr:user 0m15.092s 2026-02-20T23:02:15.160 INFO:teuthology.orchestra.run.trial032.stderr:sys 0m0.746s 2026-02-20T23:03:24.872 INFO:teuthology.orchestra.run.trial107.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2026-02-20T23:03:31.624 INFO:teuthology.orchestra.run.trial107.stderr: 93.3% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2026-02-20T23:03:54.138 INFO:teuthology.orchestra.run.trial107.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2026-02-20T23:03:54.139 INFO:teuthology.orchestra.run.trial107.stderr: 2026-02-20T23:03:54.139 INFO:teuthology.orchestra.run.trial107.stderr:real 1m44.215s 2026-02-20T23:03:54.139 INFO:teuthology.orchestra.run.trial107.stderr:user 4m46.206s 2026-02-20T23:03:54.140 INFO:teuthology.orchestra.run.trial107.stderr:sys 0m14.594s 2026-02-20T23:03:54.141 INFO:tasks.ceph:Archiving logs... 2026-02-20T23:03:54.142 DEBUG:teuthology.misc:Transferring archived files from trial032:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial032/log 2026-02-20T23:03:54.142 DEBUG:teuthology.orchestra.run.trial032:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T23:03:55.309 DEBUG:teuthology.misc:Transferring archived files from trial107:/var/log/ceph to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial107/log 2026-02-20T23:03:55.310 DEBUG:teuthology.orchestra.run.trial107:> sudo tar c -f - -C /var/log/ceph -- . 2026-02-20T23:04:16.622 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-20T23:04:16.634 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-20T23:04:16.634 DEBUG:teuthology.orchestra.run.trial032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T23:04:16.636 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-20T23:04:16.776 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-20T23:04:16.787 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-20T23:04:16.787 DEBUG:teuthology.orchestra.run.trial032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T23:04:16.789 DEBUG:teuthology.orchestra.run.trial107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout: remote refid st t when poll reach delay offset jitter 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout:============================================================================== 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout:*23.186.168.132 132.163.97.4 2 u 313 128 354 69.684 -1.379 1.173 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout:+ntp.shastacoe.n 66.244.0.68 2 u 118 128 377 76.618 +0.134 1.135 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout: 104.233.211.205 .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2026-02-20T23:04:16.837 INFO:teuthology.orchestra.run.trial032.stdout:+us-east-2.clear 130.207.244.240 2 u 99 128 357 37.039 -2.801 1.029 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout: remote refid st t when poll reach delay offset jitter 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout:============================================================================== 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout:*23.186.168.132 132.163.97.4 2 u 46 128 337 69.608 +1.324 2.757 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout:+ntp.shastacoe.n 66.244.0.68 2 u 26 128 377 76.504 +2.112 0.410 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout: 104.233.211.205 .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2026-02-20T23:04:16.838 INFO:teuthology.orchestra.run.trial107.stdout:+us-east-2.clear 130.207.244.240 2 u 26 128 373 36.945 -0.479 0.589 2026-02-20T23:04:16.838 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-20T23:04:16.854 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-20T23:04:16.855 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-20T23:04:16.868 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-20T23:04:16.880 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-20T23:04:16.896 INFO:teuthology.task.internal:Duration was 2562.939265 seconds 2026-02-20T23:04:16.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-20T23:04:16.909 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-20T23:04:16.909 DEBUG:teuthology.orchestra.run.trial032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T23:04:16.910 DEBUG:teuthology.orchestra.run.trial107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-20T23:04:16.936 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-20T23:04:16.936 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial032.front.sepia.ceph.com 2026-02-20T23:04:16.936 DEBUG:teuthology.orchestra.run.trial032:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T23:04:16.985 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial107.front.sepia.ceph.com 2026-02-20T23:04:16.985 DEBUG:teuthology.orchestra.run.trial107:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-20T23:04:16.991 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-20T23:04:16.992 DEBUG:teuthology.orchestra.run.trial032:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T23:04:17.032 DEBUG:teuthology.orchestra.run.trial107:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T23:04:17.252 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-20T23:04:17.252 DEBUG:teuthology.orchestra.run.trial032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T23:04:17.254 DEBUG:teuthology.orchestra.run.trial107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-20T23:04:17.262 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T23:04:17.263 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T23:04:17.264 INFO:teuthology.orchestra.run.trial032.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T23:04:17.264 INFO:teuthology.orchestra.run.trial032.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T23:04:17.264 INFO:teuthology.orchestra.run.trial032.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T23:04:17.265 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-20T23:04:17.265 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-20T23:04:17.265 INFO:teuthology.orchestra.run.trial107.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-20T23:04:17.266 INFO:teuthology.orchestra.run.trial107.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-20T23:04:17.266 INFO:teuthology.orchestra.run.trial107.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-20T23:04:17.330 INFO:teuthology.orchestra.run.trial032.stderr: 81.4% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T23:04:17.332 INFO:teuthology.orchestra.run.trial107.stderr: 81.5% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-20T23:04:17.334 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-20T23:04:17.355 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-20T23:04:17.356 DEBUG:teuthology.orchestra.run.trial032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T23:04:17.383 DEBUG:teuthology.orchestra.run.trial107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-20T23:04:17.392 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-20T23:04:17.413 DEBUG:teuthology.orchestra.run.trial032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T23:04:17.429 DEBUG:teuthology.orchestra.run.trial107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-20T23:04:17.435 INFO:teuthology.orchestra.run.trial032.stdout:kernel.core_pattern = core 2026-02-20T23:04:17.445 INFO:teuthology.orchestra.run.trial107.stdout:kernel.core_pattern = core 2026-02-20T23:04:17.453 DEBUG:teuthology.orchestra.run.trial032:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T23:04:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T23:04:17.491 DEBUG:teuthology.orchestra.run.trial107:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-20T23:04:17.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-20T23:04:17.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-20T23:04:17.518 INFO:teuthology.task.internal:Transferring archived files... 2026-02-20T23:04:17.519 DEBUG:teuthology.misc:Transferring archived files from trial032:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial032 2026-02-20T23:04:17.519 DEBUG:teuthology.orchestra.run.trial032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T23:04:17.577 DEBUG:teuthology.misc:Transferring archived files from trial107:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-20_16:15:01-rbd-wip-pdonnell-testing-20260219.182737-tentacle-distro-default-trial/61587/remote/trial107 2026-02-20T23:04:17.577 DEBUG:teuthology.orchestra.run.trial107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-20T23:04:17.612 INFO:teuthology.task.internal:Removing archive directory... 2026-02-20T23:04:17.612 DEBUG:teuthology.orchestra.run.trial032:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T23:04:17.614 DEBUG:teuthology.orchestra.run.trial107:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-20T23:04:17.641 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-20T23:04:17.660 INFO:teuthology.task.internal:Not uploading archives. 2026-02-20T23:04:17.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-20T23:04:17.680 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-20T23:04:17.680 DEBUG:teuthology.orchestra.run.trial032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T23:04:17.682 DEBUG:teuthology.orchestra.run.trial107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-20T23:04:17.685 INFO:teuthology.orchestra.run.trial032.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 23:04 /home/ubuntu/cephtest 2026-02-20T23:04:17.689 INFO:teuthology.orchestra.run.trial107.stdout: 1179687 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 20 23:04 /home/ubuntu/cephtest 2026-02-20T23:04:17.690 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-20T23:04:17.706 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-20T23:04:17.740 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 2562.9392652511597 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-20T23:04:17.741 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-20T23:04:17.833 INFO:teuthology.run:pass